[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:54 dist-test-slave-sf70 krb5kdc[2042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:54 dist-test-slave-sf70 krb5kdc[2042](info): set up 2 sockets
Sep 28 14:57:54 dist-test-slave-sf70 krb5kdc[2042](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:57:55.293262 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:57:55.411463 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45571
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:55.495178 2055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:55.495409 2055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:55.495445 2055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:55.496651 2055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:55.496714 2055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:55.496739 2055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:55.496771 2055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:55.496805 2055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:55.498263 2055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:55.498538 2055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:55.498764 2055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:55.504295 2062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:55.504285 2061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:55.504384 2055 server_base.cc:1034] running on GCE node
W20260928 14:57:55.504251 2064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:55.504808 2055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:55.505050 2055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:55.506217 2055 hybrid_clock.cc:648] HybridClock initialized: now 1790607475506187 us; error 49 us; skew 500 ppm
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:55.508834 2055 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:57:55.509318 2055 webserver.cc:460] Webserver started at http://127.1.252.254:40363/ using document root <none> and password file <none>
I20260928 14:57:55.509554 2055 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:55.509620 2055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:55.509773 2055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:55.510802 2055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "3449d0dcc76f43d4aef2727c084cae39"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "eb2df29831dea24bb2b5af06829f106f"
server_key_iv: "c704243839f9dfe78227f12086a14ff3"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.511164 2055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "3449d0dcc76f43d4aef2727c084cae39"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "eb2df29831dea24bb2b5af06829f106f"
server_key_iv: "c704243839f9dfe78227f12086a14ff3"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.512542 2055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:55.513283 2071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.513466 2055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:55.513571 2055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "3449d0dcc76f43d4aef2727c084cae39"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "eb2df29831dea24bb2b5af06829f106f"
server_key_iv: "c704243839f9dfe78227f12086a14ff3"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.513644 2055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:55.546828 2055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:55.549363 2055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:55.549539 2055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:55.553686 2055 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:45571
I20260928 14:57:55.553725 2123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:45571 every 8 connection(s)
I20260928 14:57:55.554080 2055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:55.554909 2124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:55.556479 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2055
I20260928 14:57:55.556591 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:57:55.557384 2035 external_mini_cluster.cc:1428] Setting key c107d8b21bf48861989f852ca8b53a45
I20260928 14:57:55.557652 2124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39: Bootstrap starting.
I20260928 14:57:55.558351 2124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:55.558699 2124 log.cc:824] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:55.559437 2124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39: No bootstrap required, opened a new log
I20260928 14:57:55.561682 2124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3449d0dcc76f43d4aef2727c084cae39" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45571 } }
I20260928 14:57:55.561844 2124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:55.561882 2124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3449d0dcc76f43d4aef2727c084cae39, State: Initialized, Role: FOLLOWER
I20260928 14:57:55.562011 2124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3449d0dcc76f43d4aef2727c084cae39" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45571 } }
I20260928 14:57:55.562094 2124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:55.562131 2124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:55.562178 2124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:55.563302 2124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3449d0dcc76f43d4aef2727c084cae39" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45571 } }
I20260928 14:57:55.563426 2124 leader_election.cc:302] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3449d0dcc76f43d4aef2727c084cae39; no voters:
I20260928 14:57:55.563592 2124 leader_election.cc:288] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:55.563616 2129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:55.563786 2129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [term 1 LEADER]: Becoming Leader. State: Replica: 3449d0dcc76f43d4aef2727c084cae39, State: Running, Role: LEADER
I20260928 14:57:55.563838 2124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:55.563910 2129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3449d0dcc76f43d4aef2727c084cae39" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45571 } }
I20260928 14:57:55.565544 2130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3449d0dcc76f43d4aef2727c084cae39. Latest consensus state: current_term: 1 leader_uuid: "3449d0dcc76f43d4aef2727c084cae39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3449d0dcc76f43d4aef2727c084cae39" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45571 } } }
I20260928 14:57:55.565639 2130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [sys.catalog]: This master's current role is: LEADER
W20260928 14:57:55.565927 2143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:55.566284 2143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:55.566036 2131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3449d0dcc76f43d4aef2727c084cae39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3449d0dcc76f43d4aef2727c084cae39" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45571 } } }
I20260928 14:57:55.566354 2131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:55.566424 2145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:55.566614 2145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:55.568145 2145 catalog_manager.cc:1347] Generated new cluster ID: 89d86bf9b99f4950af6642e12155b1a3
I20260928 14:57:55.568199 2145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:55.575839 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:55.558291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.1.252.254:45571)
0928 14:57:55.558996 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:55.559005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:55.559041 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:57:55.561670 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:55.561677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:55.561724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:55.561875 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:55.564042 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.565033 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.566540 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.567029 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.567156 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:55.571315 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:55.571318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:55.571319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:55.571329 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:55.573850 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:55.574352 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:55.574354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:55.574355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:55.574470 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:55.574765 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:55.574766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:55.574766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:55.575049 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:55.575200 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:55.575499 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 14:57:55.575624 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":88,"threads_started":1}
I20260928 14:57:55.579841 2145 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:55.580538 2145 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:55.584165 2145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3449d0dcc76f43d4aef2727c084cae39: Generated new TSK 0
I20260928 14:57:55.584326 2145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:57:55.611235 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45571
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:55.692608 2150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:55.692836 2150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:55.692878 2150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:55.694033 2150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:55.694097 2150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:55.694144 2150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:57:55.695670 2150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:55.695997 2150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:55.696233 2150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:55.701076 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:55.701103 2158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:55.701103 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:55.701429 2150 server_base.cc:1034] running on GCE node
I20260928 14:57:55.701630 2150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:55.701861 2150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:55.703032 2150 hybrid_clock.cc:648] HybridClock initialized: now 1790607475703029 us; error 38 us; skew 500 ppm
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:55.705435 2150 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:57:55.705891 2150 webserver.cc:460] Webserver started at http://127.1.252.193:42003/ using document root <none> and password file <none>
I20260928 14:57:55.706144 2150 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:55.706212 2150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:55.706346 2150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:55.707355 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "0f5e46c211dc41279dc8a059e752de35"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "f977b045d304f8218e6f2ec085f3cb33"
server_key_iv: "4fd94d0d82b0288077f8773159d4e143"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.707715 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "0f5e46c211dc41279dc8a059e752de35"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "f977b045d304f8218e6f2ec085f3cb33"
server_key_iv: "4fd94d0d82b0288077f8773159d4e143"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.709079 2150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:57:55.709813 2165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.709987 2150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.710083 2150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "0f5e46c211dc41279dc8a059e752de35"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "f977b045d304f8218e6f2ec085f3cb33"
server_key_iv: "4fd94d0d82b0288077f8773159d4e143"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.710170 2150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:55.742308 2150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:55.744498 2150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:55.744664 2150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:55.744931 2150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:55.745249 2150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:55.745302 2150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.745347 2150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:55.745374 2150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.751333 2150 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:46063
I20260928 14:57:55.751389 2279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:46063 every 8 connection(s)
I20260928 14:57:55.751688 2150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:57:55.756453 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2150
I20260928 14:57:55.756537 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:55.756685 2035 external_mini_cluster.cc:1428] Setting key d35d9a6ff92ed20ba44504eaafd9e119
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:55.761497 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:55.752367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45571 (local address 127.1.252.193:47729)
0928 14:57:55.752532 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:55.752575 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:55.753087 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:55.753398 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:55.753402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:55.753953 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:55.754142 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:55.754146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.754938 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.754939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:55.755941 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:55.755946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.756398 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.756400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:55.756562 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:55.757124 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:55.757139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:55.758498 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:55.760134 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:55.760141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:55.760143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:55.760386 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:55.760640 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:55.760640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:55.760640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:55.760740 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:55.761025 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:55.761030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:55.761137 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 14:57:55.761217 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:57:55.761333 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260928 14:57:55.761510 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:55.752399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:47729 (local address 127.1.252.254:45571)
0928 14:57:55.752448 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:55.752450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:55.752660 (+ 210us) server_negotiation.cc:407] Connection header received
0928 14:57:55.753211 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:55.753212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:55.753229 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:55.753294 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:55.754263 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.754838 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.756031 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.756301 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.756375 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:55.758603 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:55.758605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:55.758605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:55.758612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:55.760040 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:55.760488 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:55.760490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:55.760490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:55.760565 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:55.760805 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:55.760806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:55.760807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:55.760957 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:55.761032 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:55.761270 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 14:57:55.761371 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:57:55.762338 2280 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45571
I20260928 14:57:55.762461 2280 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:55.762630 2280 heartbeater.cc:511] Master 127.1.252.254:45571 requested a full tablet report, sending...
I20260928 14:57:55.763181 2088 ts_manager.cc:194] Registered new tserver with Master: 0f5e46c211dc41279dc8a059e752de35 (127.1.252.193:46063)
I20260928 14:57:55.763823 2088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:47729
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:57:55.782685 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45571
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:55.866343 2285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:55.866613 2285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:55.866653 2285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:55.867806 2285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:55.867869 2285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:55.867936 2285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:57:55.869385 2285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:55.869647 2285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:55.869870 2285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:55.874912 2291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:55.874940 2290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:55.875020 2285 server_base.cc:1034] running on GCE node
W20260928 14:57:55.874939 2293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:55.875324 2285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:55.875517 2285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:55.876673 2285 hybrid_clock.cc:648] HybridClock initialized: now 1790607475876658 us; error 43 us; skew 500 ppm
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:55.879530 2285 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:57:55.880059 2285 webserver.cc:460] Webserver started at http://127.1.252.194:36197/ using document root <none> and password file <none>
I20260928 14:57:55.880290 2285 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:55.880358 2285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:55.880491 2285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:55.881515 2285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "6f3c8e3840844d8e9b8fa11bff5ab062"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "840f91fc48cc264aac6dda4a0c6db1a4"
server_key_iv: "fb933f15f6b7d0fc33e34231548351fa"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.881882 2285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "6f3c8e3840844d8e9b8fa11bff5ab062"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "840f91fc48cc264aac6dda4a0c6db1a4"
server_key_iv: "fb933f15f6b7d0fc33e34231548351fa"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.883154 2285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:57:55.883828 2300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.883982 2285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:57:55.884086 2285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "6f3c8e3840844d8e9b8fa11bff5ab062"
format_stamp: "Formatted at 2026-09-28 14:57:55 on dist-test-slave-sf70"
server_key: "840f91fc48cc264aac6dda4a0c6db1a4"
server_key_iv: "fb933f15f6b7d0fc33e34231548351fa"
server_key_version: "encryptionkey@0"
I20260928 14:57:55.884173 2285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:55.916667 2285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:55.918731 2285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:55.918910 2285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:55.919167 2285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:55.919493 2285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:55.919552 2285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.919601 2285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:55.919636 2285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:55.926272 2285 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:41675
I20260928 14:57:55.926329 2414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:41675 every 8 connection(s)
I20260928 14:57:55.926648 2285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:55.927374 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2285
I20260928 14:57:55.927508 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:55.927724 2035 external_mini_cluster.cc:1428] Setting key ae25bbd662e60c608647f06026479b8e
Sep 28 14:57:55 dist-test-slave-sf70 krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607475, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
I20260928 14:57:55.937557 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:55.927397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:49691 (local address 127.1.252.254:45571)
0928 14:57:55.927468 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:55.927471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:55.927700 (+ 229us) server_negotiation.cc:407] Connection header received
0928 14:57:55.928283 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:55.928284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:55.928297 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:55.928398 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:55.929723 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.930577 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.931571 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.931989 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.932068 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:55.934685 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:55.934687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:55.934687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:55.934693 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:55.936160 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:55.936664 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:55.936665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:55.936665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:55.936729 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:55.937026 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:55.937027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:55.937027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:55.937129 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:55.937181 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:55.937433 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:57:55.937493 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:57:55.937855 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:55.927343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45571 (local address 127.1.252.194:49691)
0928 14:57:55.927563 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:55.927605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:55.928192 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:55.928495 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:55.928499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:55.929333 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:55.929609 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:55.929615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.930680 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.930681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:55.931476 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:55.931478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:55.932072 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:55.932073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:55.932229 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:55.932757 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:55.932771 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:55.934589 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:55.936244 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:55.936252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:55.936254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:55.936562 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:55.936796 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:55.936798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:55.936798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:55.936954 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:55.937191 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:55.937196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:55.937317 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 14:57:55.937407 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:57:55.937533 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260928 14:57:55.938417 2415 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45571
I20260928 14:57:55.938526 2415 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:55.938746 2415 heartbeater.cc:511] Master 127.1.252.254:45571 requested a full tablet report, sending...
I20260928 14:57:55.939158 2088 ts_manager.cc:194] Registered new tserver with Master: 6f3c8e3840844d8e9b8fa11bff5ab062 (127.1.252.194:41675)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
I20260928 14:57:55.939558 2088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:49691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:57:55.955726 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45571
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:56.037184 2420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:56.037377 2420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:56.037400 2420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:56.038534 2420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:56.038604 2420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:56.038631 2420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:57:56.040141 2420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:56.040385 2420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:56.040614 2420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:56.045464 2426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:56.045516 2428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:56.045647 2420 server_base.cc:1034] running on GCE node
W20260928 14:57:56.045472 2425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:56.045890 2420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:56.046123 2420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:56.047276 2420 hybrid_clock.cc:648] HybridClock initialized: now 1790607476047273 us; error 35 us; skew 500 ppm
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:56.049779 2420 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:57:56.050212 2420 webserver.cc:460] Webserver started at http://127.1.252.195:42357/ using document root <none> and password file <none>
I20260928 14:57:56.050436 2420 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:56.050500 2420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:56.050644 2420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:56.051719 2420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "ea7a384f02b04984a7a36cf4d0c94263"
format_stamp: "Formatted at 2026-09-28 14:57:56 on dist-test-slave-sf70"
server_key: "cf42c393c6448f96b3c76b460b03f9b8"
server_key_iv: "c3e939a6b3375589bdb155c24f34b05f"
server_key_version: "encryptionkey@0"
I20260928 14:57:56.052095 2420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "ea7a384f02b04984a7a36cf4d0c94263"
format_stamp: "Formatted at 2026-09-28 14:57:56 on dist-test-slave-sf70"
server_key: "cf42c393c6448f96b3c76b460b03f9b8"
server_key_iv: "c3e939a6b3375589bdb155c24f34b05f"
server_key_version: "encryptionkey@0"
I20260928 14:57:56.053378 2420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:56.054080 2435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:56.054262 2420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:56.054356 2420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "ea7a384f02b04984a7a36cf4d0c94263"
format_stamp: "Formatted at 2026-09-28 14:57:56 on dist-test-slave-sf70"
server_key: "cf42c393c6448f96b3c76b460b03f9b8"
server_key_iv: "c3e939a6b3375589bdb155c24f34b05f"
server_key_version: "encryptionkey@0"
I20260928 14:57:56.054431 2420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:56.067555 2420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:56.069288 2420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:56.069439 2420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:56.069644 2420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:56.069936 2420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:56.069988 2420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:56.070027 2420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:56.070067 2420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:56.075734 2420 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:33327
I20260928 14:57:56.075822 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:33327 every 8 connection(s)
I20260928 14:57:56.076124 2420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:56.080200 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2420
I20260928 14:57:56.080309 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:56.080488 2035 external_mini_cluster.cc:1428] Setting key e568e9b9ec6ea5bc99ed416c2129d392
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:56.085139 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:56.076829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:52573 (local address 127.1.252.254:45571)
0928 14:57:56.076885 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:56.076887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:56.077088 (+ 201us) server_negotiation.cc:407] Connection header received
0928 14:57:56.077550 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:56.077551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:56.077560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:56.077618 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:56.078459 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.078942 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.079843 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.080137 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.080215 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:56.082507 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:56.082508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:56.082508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:56.082515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:56.083777 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:56.084248 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:56.084249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:56.084250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:56.084317 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:56.084562 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:56.084563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:56.084563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:56.084707 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:56.084760 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:56.085010 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 14:57:56.085077 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:57:56.085301 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:56.076815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45571 (local address 127.1.252.195:52573)
0928 14:57:56.076969 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:56.077008 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:56.077471 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:56.077668 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:56.077671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:56.078209 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:56.078388 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:56.078393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.079016 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.079017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:56.079772 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:56.079773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.080218 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.080219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:56.080363 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:56.080864 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:56.080880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:56.082418 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:56.083848 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:56.083853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:56.083854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:56.084166 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:56.084387 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:56.084388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:56.084388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:56.084482 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:56.084767 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:56.084771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:56.084876 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 14:57:56.084984 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:57:56.085093 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 14:57:56.085785 2549 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45571
I20260928 14:57:56.085857 2549 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:56.086061 2549 heartbeater.cc:511] Master 127.1.252.254:45571 requested a full tablet report, sending...
I20260928 14:57:56.086416 2088 ts_manager.cc:194] Registered new tserver with Master: ea7a384f02b04984a7a36cf4d0c94263 (127.1.252.195:33327)
I20260928 14:57:56.086794 2088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:52573
I20260928 14:57:56.092540 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:57:56.114955 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:56.107216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.1.252.193:46063)
0928 14:57:56.107461 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:56.107465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:56.107475 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:57:56.107501 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:56.107502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:56.107517 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:56.107581 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:56.108186 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.108896 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.109592 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.109900 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.109967 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:56.112077 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:56.112085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:56.112086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:56.112094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:56.113746 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:56.114118 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:56.114121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:56.114122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:56.114173 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:56.114373 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:56.114374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:56.114374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:56.114513 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:56.114674 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:56.114744 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:57:56.114821 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":166,"threads_started":1}
W20260928 14:57:56.115301 2193 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37962
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:57:56.136445 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:56.129746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.1.252.193:46063)
0928 14:57:56.129831 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:56.129834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:56.129877 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:57:56.129992 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:56.129993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:56.130005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:56.130078 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:56.130636 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.131122 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.131919 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:56.132124 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:56.132202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:56.134201 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:56.134202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:56.134203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:56.134210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:56.135367 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:56.135693 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:56.135695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:56.135695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:56.135751 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:56.135953 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:56.135954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:56.135955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:56.136054 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:56.136173 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:56.136267 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:57:56.136342 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:57:56.137050 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2150
I20260928 14:57:56.141582 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2285
I20260928 14:57:56.145452 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2420
I20260928 14:57:56.149658 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2055
2026-09-28T14:57:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1925 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2563](info): set up 2 sockets
Sep 28 14:57:56 dist-test-slave-sf70 krb5kdc[2563](info): commencing operation
krb5kdc: starting...
W20260928 14:57:57.197054 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:57:57.315682 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46365
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:57.395612 2576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:57.395859 2576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:57.395900 2576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:57.397009 2576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:57.397080 2576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:57.397104 2576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:57.397126 2576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:57.397156 2576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:57.398608 2576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:57.398881 2576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:57.399108 2576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:57.404079 2581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.404076 2582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.404258 2584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:57.404304 2576 server_base.cc:1034] running on GCE node
I20260928 14:57:57.404546 2576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:57.404781 2576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:57.405921 2576 hybrid_clock.cc:648] HybridClock initialized: now 1790607477405910 us; error 30 us; skew 500 ppm
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:57.408491 2576 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:57:57.408980 2576 webserver.cc:460] Webserver started at http://127.1.252.254:33757/ using document root <none> and password file <none>
I20260928 14:57:57.409227 2576 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:57.409303 2576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:57.409461 2576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:57.410498 2576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "abe4523420964df6a8a05bd5131abe85"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "1dcf3d1f030a21ce13bf56b0d590a6aa"
server_key_iv: "7c2103dd8f72e8ec44de6d4c55dd9c4f"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.410856 2576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "abe4523420964df6a8a05bd5131abe85"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "1dcf3d1f030a21ce13bf56b0d590a6aa"
server_key_iv: "7c2103dd8f72e8ec44de6d4c55dd9c4f"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.412201 2576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:57.412936 2591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.413106 2576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:57.413219 2576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "abe4523420964df6a8a05bd5131abe85"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "1dcf3d1f030a21ce13bf56b0d590a6aa"
server_key_iv: "7c2103dd8f72e8ec44de6d4c55dd9c4f"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.413302 2576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:57.424057 2576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:57.425918 2576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:57.426071 2576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:57.430070 2576 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:46365
I20260928 14:57:57.430106 2643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:46365 every 8 connection(s)
I20260928 14:57:57.430452 2576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:57.431275 2644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:57.433547 2644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85: Bootstrap starting.
I20260928 14:57:57.434134 2644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:57.434432 2644 log.cc:824] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:57.435133 2644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85: No bootstrap required, opened a new log
I20260928 14:57:57.436705 2644 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe4523420964df6a8a05bd5131abe85" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46365 } }
I20260928 14:57:57.436837 2644 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:57.436859 2644 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe4523420964df6a8a05bd5131abe85, State: Initialized, Role: FOLLOWER
I20260928 14:57:57.437036 2644 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe4523420964df6a8a05bd5131abe85" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46365 } }
I20260928 14:57:57.437099 2644 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:57.437120 2644 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:57.437182 2644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:57.437690 2644 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe4523420964df6a8a05bd5131abe85" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46365 } }
I20260928 14:57:57.437798 2644 leader_election.cc:302] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe4523420964df6a8a05bd5131abe85; no voters:
I20260928 14:57:57.437974 2644 leader_election.cc:288] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:57.438019 2647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:57.438207 2647 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [term 1 LEADER]: Becoming Leader. State: Replica: abe4523420964df6a8a05bd5131abe85, State: Running, Role: LEADER
I20260928 14:57:57.438231 2644 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:57.438297 2647 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe4523420964df6a8a05bd5131abe85" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46365 } }
I20260928 14:57:57.438697 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe4523420964df6a8a05bd5131abe85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe4523420964df6a8a05bd5131abe85" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46365 } } }
I20260928 14:57:57.438705 2649 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe4523420964df6a8a05bd5131abe85. Latest consensus state: current_term: 1 leader_uuid: "abe4523420964df6a8a05bd5131abe85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe4523420964df6a8a05bd5131abe85" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46365 } } }
I20260928 14:57:57.438807 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:57.438814 2649 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:57.439009 2651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:57.439551 2651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:57.439992 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2576
I20260928 14:57:57.440075 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:57:57.440229 2035 external_mini_cluster.cc:1428] Setting key 37e5173529200be439957c9affba8c80
I20260928 14:57:57.441992 2651 catalog_manager.cc:1347] Generated new cluster ID: b9f74988074c4aafb099cee169188a34
I20260928 14:57:57.442049 2651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:57.451951 2651 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:57.452147 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.440721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.1.252.254:46365)
0928 14:57:57.440926 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.440936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.441137 (+ 201us) server_negotiation.cc:407] Connection header received
0928 14:57:57.441256 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.441263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.441315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.441476 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:57.442141 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.443561 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.444603 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.445271 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.445412 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.448184 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:57.448188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:57.448188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:57.448201 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:57.450120 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.450525 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.450528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.450528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.450628 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.450852 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.450853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.450854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.451118 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:57.451255 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.451648 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.451816 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260928 14:57:57.452615 2651 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:57.457522 2651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe4523420964df6a8a05bd5131abe85: Generated new TSK 0
I20260928 14:57:57.457705 2651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:57:57.488097 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46365
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:57.598868 2670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:57.599102 2670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:57.599129 2670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:57.600602 2670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:57.600649 2670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:57.600682 2670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:57:57.602396 2670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:57.602677 2670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:57.602974 2670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:57.609159 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.609258 2678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.609218 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:57.609338 2670 server_base.cc:1034] running on GCE node
I20260928 14:57:57.609542 2670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:57.609755 2670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:57.610909 2670 hybrid_clock.cc:648] HybridClock initialized: now 1790607477610893 us; error 29 us; skew 500 ppm
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:57.613461 2670 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:57:57.613907 2670 webserver.cc:460] Webserver started at http://127.1.252.193:46003/ using document root <none> and password file <none>
I20260928 14:57:57.614147 2670 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:57.614213 2670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:57.614368 2670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:57.615414 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "f6f0fbf2512cffa278472a94c5a3bb08"
server_key_iv: "37045175eb6e627f80c7b85423aaa3c2"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.615855 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "f6f0fbf2512cffa278472a94c5a3bb08"
server_key_iv: "37045175eb6e627f80c7b85423aaa3c2"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.617205 2670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:57.617914 2685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.618084 2670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:57.618181 2670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "f6f0fbf2512cffa278472a94c5a3bb08"
server_key_iv: "37045175eb6e627f80c7b85423aaa3c2"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.618259 2670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:57.647737 2670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:57.649616 2670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:57.649786 2670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:57.650033 2670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:57.650344 2670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:57.650396 2670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.650446 2670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:57.650475 2670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.656512 2670 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:43233
I20260928 14:57:57.656558 2799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:43233 every 8 connection(s)
I20260928 14:57:57.656898 2670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:57.663808 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2670
I20260928 14:57:57.663938 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:57.664135 2035 external_mini_cluster.cc:1428] Setting key dcdad1d87b06d588526d00beef899122
I20260928 14:57:57.666637 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.657620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:45271 (local address 127.1.252.254:46365)
0928 14:57:57.657685 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.657687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.657956 (+ 269us) server_negotiation.cc:407] Connection header received
0928 14:57:57.658858 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.658859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.658869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.658922 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:57.659525 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.660045 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.661023 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.661228 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.661353 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.663438 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:57.663439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:57.663439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:57.663446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:57.665154 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.665575 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.665576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.665577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.665649 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.665918 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.665919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.665919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.666063 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:57.666141 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.666463 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.666550 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:57:57.666847 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.657607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:46365 (local address 127.1.252.193:45271)
0928 14:57:57.657793 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:57.657828 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:57.658751 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:57.658972 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:57.658975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:57.659204 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:57.659404 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.659409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.660127 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.660128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.660931 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.660934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.661366 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.661367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.661480 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.661922 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:57.661930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:57.663350 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:57.665242 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:57.665248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:57.665249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:57.665499 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:57.665721 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:57.665722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:57.665722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:57.665850 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:57.666165 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:57.666171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:57.666290 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:57:57.666432 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 14:57:57.666603 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260928 14:57:57.667434 2800 heartbeater.cc:347] Connected to a master server at 127.1.252.254:46365
I20260928 14:57:57.667550 2800 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:57.667770 2800 heartbeater.cc:511] Master 127.1.252.254:46365 requested a full tablet report, sending...
I20260928 14:57:57.668337 2608 ts_manager.cc:194] Registered new tserver with Master: 01c770bbc5cf4e1cb5907b7b8529bf6f (127.1.252.193:43233)
I20260928 14:57:57.668946 2608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:45271
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:57:57.691306 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46365
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:57.772464 2805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:57.772658 2805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:57.772681 2805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:57.773737 2805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:57.773770 2805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:57.773795 2805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:57:57.775318 2805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:57.775540 2805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:57.775723 2805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:57.780544 2813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.780552 2811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.780561 2810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:57.781014 2805 server_base.cc:1034] running on GCE node
I20260928 14:57:57.781195 2805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:57.781385 2805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:57.782533 2805 hybrid_clock.cc:648] HybridClock initialized: now 1790607477782518 us; error 27 us; skew 500 ppm
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:57.784876 2805 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:57:57.785351 2805 webserver.cc:460] Webserver started at http://127.1.252.194:36861/ using document root <none> and password file <none>
I20260928 14:57:57.785622 2805 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:57.785688 2805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:57.785837 2805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:57.786878 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "4c02332a6cfe4c88911000e663962401"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "f4273276370a196bbafc694960abd1ff"
server_key_iv: "dc6db9785ccfe4a8fa6df183ead3c1c1"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.787240 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "4c02332a6cfe4c88911000e663962401"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "f4273276370a196bbafc694960abd1ff"
server_key_iv: "dc6db9785ccfe4a8fa6df183ead3c1c1"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.788631 2805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:57.789333 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.789516 2805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:57.789613 2805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "4c02332a6cfe4c88911000e663962401"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "f4273276370a196bbafc694960abd1ff"
server_key_iv: "dc6db9785ccfe4a8fa6df183ead3c1c1"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.789693 2805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:57.799746 2805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:57.801577 2805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:57.801733 2805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:57.801946 2805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:57.802233 2805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:57.802286 2805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.802325 2805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:57.802351 2805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.808468 2805 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:43609
I20260928 14:57:57.808517 2934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:43609 every 8 connection(s)
I20260928 14:57:57.808804 2805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:57.816090 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2805
I20260928 14:57:57.816217 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:57.816433 2035 external_mini_cluster.cc:1428] Setting key de0d185c1d20334190d643634a81fbd5
I20260928 14:57:57.818224 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.809471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:32835 (local address 127.1.252.254:46365)
0928 14:57:57.809512 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.809514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.809724 (+ 210us) server_negotiation.cc:407] Connection header received
0928 14:57:57.810672 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.810674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.810686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.810737 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:57.811348 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.811854 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.812869 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.813063 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.813146 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.815243 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:57.815244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:57.815244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:57.815251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:57.816842 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.817252 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.817253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.817253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.817314 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.817568 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.817569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.817570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.817683 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:57.817758 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.818057 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.818138 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:57:57.818398 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.809428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:46365 (local address 127.1.252.194:32835)
0928 14:57:57.809609 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:57.809643 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:57.810552 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:57.810805 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:57.810809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:57.811048 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:57.811254 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.811258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.811993 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.811994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.812771 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.812772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.813155 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.813156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.813270 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.813778 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:57.813786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:57.815151 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:57.816938 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:57.816945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:57.816947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:57.817161 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:57.817381 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:57.817382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:57.817383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:57.817492 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:57.817749 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:57.817755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:57.817910 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 14:57:57.818023 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:57:57.818170 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260928 14:57:57.818950 2935 heartbeater.cc:347] Connected to a master server at 127.1.252.254:46365
I20260928 14:57:57.819067 2935 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:57.819235 2935 heartbeater.cc:511] Master 127.1.252.254:46365 requested a full tablet report, sending...
I20260928 14:57:57.819603 2608 ts_manager.cc:194] Registered new tserver with Master: 4c02332a6cfe4c88911000e663962401 (127.1.252.194:43609)
I20260928 14:57:57.820042 2608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:32835
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:57:57.842849 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46365
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:57.926033 2940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:57.926268 2940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:57.926308 2940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:57.927578 2940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:57.927642 2940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:57.927687 2940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:57:57.929279 2940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:57.929566 2940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:57.929802 2940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:57.934898 2947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:57.934937 2946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:57.935015 2940 server_base.cc:1034] running on GCE node
W20260928 14:57:57.934907 2949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:57.935343 2940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:57.935581 2940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:57.936739 2940 hybrid_clock.cc:648] HybridClock initialized: now 1790607477936740 us; error 48 us; skew 500 ppm
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:57.939388 2940 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:57:57.939954 2940 webserver.cc:460] Webserver started at http://127.1.252.195:41259/ using document root <none> and password file <none>
I20260928 14:57:57.940212 2940 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:57.940258 2940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:57.940347 2940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:57.941325 2940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "3539ddcfce2e411d9c2dfb963653c33e"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "b172dca4e57ee827c74e475f5834a0a3"
server_key_iv: "fcdcc6d2b564e03a6dc095e13d3f04ff"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.941617 2940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "3539ddcfce2e411d9c2dfb963653c33e"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "b172dca4e57ee827c74e475f5834a0a3"
server_key_iv: "fcdcc6d2b564e03a6dc095e13d3f04ff"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.942867 2940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:57.943508 2956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.943674 2940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:57.943775 2940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "3539ddcfce2e411d9c2dfb963653c33e"
format_stamp: "Formatted at 2026-09-28 14:57:57 on dist-test-slave-sf70"
server_key: "b172dca4e57ee827c74e475f5834a0a3"
server_key_iv: "fcdcc6d2b564e03a6dc095e13d3f04ff"
server_key_version: "encryptionkey@0"
I20260928 14:57:57.943915 2940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:57.955809 2940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:57.957616 2940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:57.957758 2940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:57.957978 2940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:57:57.958257 2940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:57.958288 2940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.958308 2940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:57.958323 2940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:57.964217 2940 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35001
I20260928 14:57:57.964290 3069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35001 every 8 connection(s)
I20260928 14:57:57.964589 2940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:57.967424 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 2940
I20260928 14:57:57.967522 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:57.967660 2035 external_mini_cluster.cc:1428] Setting key 9b58f68ecf54c20ded646d75721e8a89
Sep 28 14:57:57 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:57.973953 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.965322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:60131 (local address 127.1.252.254:46365)
0928 14:57:57.965388 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.965390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.965572 (+ 182us) server_negotiation.cc:407] Connection header received
0928 14:57:57.966462 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.966462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.966472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.966526 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:57.967083 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.967790 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.968853 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.969052 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.969136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.971179 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:57.971181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:57.971182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:57.971189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:57.972585 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.973009 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.973011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.973011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.973070 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.973319 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.973320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.973321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.973436 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:57.973506 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.973778 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.973853 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:57:57.974148 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.965295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:46365 (local address 127.1.252.195:60131)
0928 14:57:57.965448 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:57.965488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:57.966372 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:57.966575 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:57.966578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:57.966827 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:57.967008 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.967011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.967919 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.967920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.968754 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.968756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.969144 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.969145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.969273 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.969788 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:57.969795 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:57.971082 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:57.972684 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:57.972690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:57.972693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:57.972919 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:57.973137 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:57.973137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:57.973138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:57.973245 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:57.973511 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:57.973517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:57.973648 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 14:57:57.973754 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:57:57.973919 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 14:57:57.974609 3070 heartbeater.cc:347] Connected to a master server at 127.1.252.254:46365
I20260928 14:57:57.974685 3070 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:57.974871 3070 heartbeater.cc:511] Master 127.1.252.254:46365 requested a full tablet report, sending...
I20260928 14:57:57.975242 2608 ts_manager.cc:194] Registered new tserver with Master: 3539ddcfce2e411d9c2dfb963653c33e (127.1.252.195:35001)
I20260928 14:57:57.975618 2608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:60131
I20260928 14:57:57.979163 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:57.986452 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.980756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.1.252.254:46365)
0928 14:57:57.980839 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.980841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.980903 (+ 62us) server_negotiation.cc:407] Connection header received
0928 14:57:57.980971 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.980971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.980981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.981053 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:57.981603 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.982068 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.982800 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.982985 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.983055 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.984201 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:57.984203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:57.984203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:57.984210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:57.985322 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.985666 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.985667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.985668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.985725 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:57.985921 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:57.985922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:57.985923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:57.986063 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:57.986169 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.986273 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.986351 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:57:57.988479 2608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:57.989972 2608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:57.996526 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.992206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43233 (local address 127.0.0.1:40416)
0928 14:57:57.992938 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:57.992946 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:57.992989 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:57.993269 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:57.993270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:57.993272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:57.993318 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.993320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.994268 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.994269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.995284 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.995287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.996228 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.996229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.996303 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.996316 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:57:57.996394 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:57:57.996446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":263,"thread_start_us":99,"threads_started":1}
I20260928 14:57:57.996567 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.992427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.1.252.193:43233)
0928 14:57:57.992573 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.992576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.993050 (+ 474us) server_negotiation.cc:407] Connection header received
0928 14:57:57.993096 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.993099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.993113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.993185 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:57.993397 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.994163 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.995391 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.996074 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.996141 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.996194 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.996429 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.996476 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 14:57:57.996896 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.992206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:43609 (local address 127.0.0.1:45014)
0928 14:57:57.992657 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:57.992685 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:57.992743 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:57.992951 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:57.992954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:57.992957 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:57.993029 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.993031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.994393 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.994394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.995682 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.995684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.996658 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.996659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.996730 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.996734 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:57.996761 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 14:57:57.996799 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":63,"thread_start_us":42,"threads_started":1}
I20260928 14:57:57.996948 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.992314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.1.252.195:35001)
0928 14:57:57.992480 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.992485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.992876 (+ 391us) server_negotiation.cc:407] Connection header received
0928 14:57:57.992916 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.992919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.992933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.993022 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:57.993286 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.994358 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.995779 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.996514 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.996577 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.996633 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.996836 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.996871 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260928 14:57:57.996961 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.992227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35001 (local address 127.0.0.1:59182)
0928 14:57:57.992814 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:57.992820 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:57.992851 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:57.993157 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:57.993158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:57.993161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:57.993203 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.993205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.994448 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.994449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.995677 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:57.995680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.996669 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.996670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:57.996735 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.996739 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:57:57.996819 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:57:57.996860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":7,"thread_start_us":100,"threads_started":1}
I20260928 14:57:57.997227 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:57.992516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.1.252.194:43609)
0928 14:57:57.992649 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:57.992657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:57.992727 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:57:57.992808 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:57.992810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:57.992822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:57.992928 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:57.993104 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.994259 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.995796 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:57.996550 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:57.997002 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:57.997071 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:57.997101 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 14:57:57.997151 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 14:57:57.997669 3004 tablet_service.cc:1467] Processing CreateTablet for tablet c93a89469a3547f4a79d6c4bf871bb6a (DEFAULT_TABLE table=test-table [id=178f7fa372d24147bf453de3ad6848b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:57.997741 2868 tablet_service.cc:1467] Processing CreateTablet for tablet c93a89469a3547f4a79d6c4bf871bb6a (DEFAULT_TABLE table=test-table [id=178f7fa372d24147bf453de3ad6848b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:57.998025 3004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93a89469a3547f4a79d6c4bf871bb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:57.997967 2733 tablet_service.cc:1467] Processing CreateTablet for tablet c93a89469a3547f4a79d6c4bf871bb6a (DEFAULT_TABLE table=test-table [id=178f7fa372d24147bf453de3ad6848b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:57.998051 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93a89469a3547f4a79d6c4bf871bb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:57.998242 2733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93a89469a3547f4a79d6c4bf871bb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:58.000666 3087 tablet_bootstrap.cc:492] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: Bootstrap starting.
I20260928 14:57:58.001389 3087 tablet_bootstrap.cc:654] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:58.001574 3089 tablet_bootstrap.cc:492] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e: Bootstrap starting.
I20260928 14:57:58.001683 3088 tablet_bootstrap.cc:492] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f: Bootstrap starting.
I20260928 14:57:58.001940 3087 log.cc:824] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:58.002255 3089 tablet_bootstrap.cc:654] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:58.002264 3088 tablet_bootstrap.cc:654] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:58.002660 3087 tablet_bootstrap.cc:492] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: No bootstrap required, opened a new log
I20260928 14:57:58.002743 3087 ts_tablet_manager.cc:1397] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:58.002905 3089 log.cc:824] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:58.002902 3088 log.cc:824] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:58.003634 3088 tablet_bootstrap.cc:492] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f: No bootstrap required, opened a new log
I20260928 14:57:58.003706 3088 ts_tablet_manager.cc:1397] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:58.003873 3089 tablet_bootstrap.cc:492] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e: No bootstrap required, opened a new log
I20260928 14:57:58.003984 3089 ts_tablet_manager.cc:1397] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:58.005121 3087 raft_consensus.cc:348] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.005298 3087 raft_consensus.cc:374] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:58.005344 3087 raft_consensus.cc:729] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c02332a6cfe4c88911000e663962401, State: Initialized, Role: FOLLOWER
I20260928 14:57:58.005466 3087 consensus_queue.cc:260] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.005744 3087 ts_tablet_manager.cc:1428] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:58.005872 2935 heartbeater.cc:503] Master 127.1.252.254:46365 was elected leader, sending a full tablet report...
I20260928 14:57:58.005748 3088 raft_consensus.cc:348] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.005924 3088 raft_consensus.cc:374] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:58.005959 3088 raft_consensus.cc:729] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c770bbc5cf4e1cb5907b7b8529bf6f, State: Initialized, Role: FOLLOWER
I20260928 14:57:58.006072 3088 consensus_queue.cc:260] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.006081 3089 raft_consensus.cc:348] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.006242 3089 raft_consensus.cc:374] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:58.006290 3089 raft_consensus.cc:729] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3539ddcfce2e411d9c2dfb963653c33e, State: Initialized, Role: FOLLOWER
I20260928 14:57:58.006331 3088 ts_tablet_manager.cc:1428] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:57:58.006443 2800 heartbeater.cc:503] Master 127.1.252.254:46365 was elected leader, sending a full tablet report...
I20260928 14:57:58.006434 3089 consensus_queue.cc:260] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.006716 3089 ts_tablet_manager.cc:1428] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:57:58.006736 3070 heartbeater.cc:503] Master 127.1.252.254:46365 was elected leader, sending a full tablet report...
W20260928 14:57:58.059337 2936 tablet.cc:2369] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:57:58.062107 3095 raft_consensus.cc:482] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:57:58.062247 3095 raft_consensus.cc:504] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.062647 3095 leader_election.cc:288] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01c770bbc5cf4e1cb5907b7b8529bf6f (127.1.252.193:43233), 4c02332a6cfe4c88911000e663962401 (127.1.252.194:43609)
I20260928 14:57:58.066531 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.062975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:56303 (local address 127.1.252.194:43609)
0928 14:57:58.063024 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:58.063026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:58.063173 (+ 147us) server_negotiation.cc:407] Connection header received
0928 14:57:58.063203 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:58.063204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:58.063216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:58.063264 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:58.063430 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.064140 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.065760 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.066170 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.066227 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.066254 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:58.066412 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:57:58.066460 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:57:58.066563 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.062862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:43609 (local address 127.1.252.195:56303)
0928 14:57:58.063076 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:58.063084 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:58.063160 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:58.063292 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:58.063293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:58.063294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:58.063359 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:58.063360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.064264 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.064265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:58.065660 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:58.065663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.066252 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.066253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:58.066331 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.066337 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:57:58.066386 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:57:58.066445 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":90,"thread_start_us":123,"threads_started":1}
I20260928 14:57:58.066586 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.062945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:33593 (local address 127.1.252.193:43233)
0928 14:57:58.063001 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:58.063003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:58.063016 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:57:58.063057 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:58.063058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:58.063070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:58.063135 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:58.063351 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.064140 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.065770 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.066173 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.066272 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.066281 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:58.066474 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 14:57:58.066517 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:57:58.066776 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.062852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43233 (local address 127.1.252.195:33593)
0928 14:57:58.062951 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:58.062963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:58.063007 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:58.063163 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:58.063164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:58.063166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:58.063244 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:58.063246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.064264 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.064265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:58.065660 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:58.065663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.066288 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.066289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:58.066368 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.066391 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:57:58.066450 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:57:58.066489 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":96}
I20260928 14:57:58.067049 2889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93a89469a3547f4a79d6c4bf871bb6a" candidate_uuid: "3539ddcfce2e411d9c2dfb963653c33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c02332a6cfe4c88911000e663962401" is_pre_election: true
I20260928 14:57:58.067147 2753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93a89469a3547f4a79d6c4bf871bb6a" candidate_uuid: "3539ddcfce2e411d9c2dfb963653c33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" is_pre_election: true
I20260928 14:57:58.067220 2889 raft_consensus.cc:2454] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3539ddcfce2e411d9c2dfb963653c33e in term 0.
I20260928 14:57:58.067324 2753 raft_consensus.cc:2454] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3539ddcfce2e411d9c2dfb963653c33e in term 0.
I20260928 14:57:58.067437 2959 leader_election.cc:302] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3539ddcfce2e411d9c2dfb963653c33e, 4c02332a6cfe4c88911000e663962401; no voters:
I20260928 14:57:58.067584 3095 raft_consensus.cc:2781] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:57:58.067667 3095 raft_consensus.cc:482] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:57:58.067703 3095 raft_consensus.cc:3037] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:58.068459 3095 raft_consensus.cc:504] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.068606 3095 leader_election.cc:288] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [CANDIDATE]: Term 1 election: Requested vote from peers 01c770bbc5cf4e1cb5907b7b8529bf6f (127.1.252.193:43233), 4c02332a6cfe4c88911000e663962401 (127.1.252.194:43609)
I20260928 14:57:58.068751 2753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93a89469a3547f4a79d6c4bf871bb6a" candidate_uuid: "3539ddcfce2e411d9c2dfb963653c33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f"
I20260928 14:57:58.068831 2753 raft_consensus.cc:3037] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:58.068828 2889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93a89469a3547f4a79d6c4bf871bb6a" candidate_uuid: "3539ddcfce2e411d9c2dfb963653c33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c02332a6cfe4c88911000e663962401"
I20260928 14:57:58.068926 2889 raft_consensus.cc:3037] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:58.069625 2753 raft_consensus.cc:2454] T c93a89469a3547f4a79d6c4bf871bb6a P 01c770bbc5cf4e1cb5907b7b8529bf6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3539ddcfce2e411d9c2dfb963653c33e in term 1.
I20260928 14:57:58.069762 2889 raft_consensus.cc:2454] T c93a89469a3547f4a79d6c4bf871bb6a P 4c02332a6cfe4c88911000e663962401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3539ddcfce2e411d9c2dfb963653c33e in term 1.
I20260928 14:57:58.069797 2960 leader_election.cc:302] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c770bbc5cf4e1cb5907b7b8529bf6f, 3539ddcfce2e411d9c2dfb963653c33e; no voters:
I20260928 14:57:58.069891 3095 raft_consensus.cc:2781] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:58.070073 3095 raft_consensus.cc:686] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [term 1 LEADER]: Becoming Leader. State: Replica: 3539ddcfce2e411d9c2dfb963653c33e, State: Running, Role: LEADER
I20260928 14:57:58.070175 3095 consensus_queue.cc:237] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } }
I20260928 14:57:58.071065 2607 catalog_manager.cc:5495] T c93a89469a3547f4a79d6c4bf871bb6a P 3539ddcfce2e411d9c2dfb963653c33e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3539ddcfce2e411d9c2dfb963653c33e (127.1.252.195). New cstate: current_term: 1 leader_uuid: "3539ddcfce2e411d9c2dfb963653c33e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c770bbc5cf4e1cb5907b7b8529bf6f" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c02332a6cfe4c88911000e663962401" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 43609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3539ddcfce2e411d9c2dfb963653c33e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35001 } health_report { overall_health: HEALTHY } } }
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:57:58.092696 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.085343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.1.252.193:43233)
0928 14:57:58.085440 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:58.085442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:58.085519 (+ 77us) server_negotiation.cc:407] Connection header received
0928 14:57:58.085612 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:58.085614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:58.085624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:58.085681 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:58.086263 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.086761 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.087488 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.087701 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.087786 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.090119 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:58.090128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:58.090129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:58.090135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:58.091550 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:58.091926 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:58.091929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:58.091930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:58.092013 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:58.092192 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:58.092193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:58.092193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:58.092320 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:58.092440 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:58.092526 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:57:58.092628 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607478, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607478, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:57:58.114085 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.107776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.1.252.193:43233)
0928 14:57:58.107854 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:58.107856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:58.107911 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:57:58.108007 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:58.108008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:58.108016 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:58.108065 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:58.108558 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.109006 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.109658 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.109831 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.109877 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.112017 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:58.112018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:58.112019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:58.112025 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:58.113179 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:58.113481 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:58.113482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:58.113482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:58.113529 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:58.113693 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:58.113693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:58.113693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:58.113778 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:58.113846 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:58.113964 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:57:58.114019 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:57:58.135609 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:58.128880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.1.252.193:43233)
0928 14:57:58.128942 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:58.128944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:58.128970 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:57:58.129038 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:58.129038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:58.129047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:58.129094 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:58.129601 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.130095 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.130868 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:58.131058 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:58.131121 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:58.133286 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:58.133287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:58.133288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:58.133294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:58.134431 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:58.134863 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:58.134864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:58.134864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:58.134918 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:58.135121 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:58.135122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:58.135122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:58.135210 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:58.135285 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:58.135442 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:57:58.135501 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 14:57:58.136137 2713 tablet_service.cc:2908] Rejecting scan request for tablet c93a89469a3547f4a79d6c4bf871bb6a: Uninitialized: clean time has not yet been initialized
I20260928 14:57:58.137002 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2670
I20260928 14:57:58.142518 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2805
I20260928 14:57:58.147121 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2940
I20260928 14:57:58.151554 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 2576
2026-09-28T14:57:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2002 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[3112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[3112](info): set up 2 sockets
Sep 28 14:57:58 dist-test-slave-sf70 krb5kdc[3112](info): commencing operation
krb5kdc: starting...
W20260928 14:57:59.181085 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:57:59.299304 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:43711
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:59.380988 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:59.381217 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:59.381254 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:59.382362 3125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:57:59.382441 3125 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:57:59.382467 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:59.382499 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:59.382524 3125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:57:59.382551 3125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:57:59.384064 3125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:43711
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:59.384334 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:59.384568 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:59.389428 3133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.389485 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:59.389637 3125 server_base.cc:1034] running on GCE node
W20260928 14:57:59.389446 3131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:59.389930 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:59.390167 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:59.391310 3125 hybrid_clock.cc:648] HybridClock initialized: now 1790607479391284 us; error 41 us; skew 500 ppm
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:59.400511 3125 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:57:59.401007 3125 webserver.cc:460] Webserver started at http://127.1.252.254:44559/ using document root <none> and password file <none>
I20260928 14:57:59.401274 3125 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:59.401322 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:59.401475 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:59.402585 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "d5c535175a1e416d80b8c7c2a36ab2e3"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "54717e18f66eb820ff200ac5d60238ae"
server_key_iv: "60d3ad11162491946bdea1f34090769a"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.402948 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "d5c535175a1e416d80b8c7c2a36ab2e3"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "54717e18f66eb820ff200ac5d60238ae"
server_key_iv: "60d3ad11162491946bdea1f34090769a"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.404342 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:57:59.405104 3140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.405299 3125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:57:59.405376 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "d5c535175a1e416d80b8c7c2a36ab2e3"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "54717e18f66eb820ff200ac5d60238ae"
server_key_iv: "60d3ad11162491946bdea1f34090769a"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.405468 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:59.425150 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:59.427490 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:59.427654 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:59.432909 3125 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:43711
I20260928 14:57:59.432978 3202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:43711 every 8 connection(s)
I20260928 14:57:59.433290 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:57:59.433936 3203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:59.434239 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3125
I20260928 14:57:59.434456 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:57:59.434633 2035 external_mini_cluster.cc:1428] Setting key 7e5b5432dc44920ad50a20effc281284
I20260928 14:57:59.436970 3203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: Bootstrap starting.
I20260928 14:57:59.437644 3203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:59.437958 3203 log.cc:824] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:59.438766 3203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: No bootstrap required, opened a new log
I20260928 14:57:59.440941 3203 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43711 } }
I20260928 14:57:59.441078 3203 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:59.441102 3203 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c535175a1e416d80b8c7c2a36ab2e3, State: Initialized, Role: FOLLOWER
I20260928 14:57:59.441233 3203 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43711 } }
I20260928 14:57:59.441309 3203 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:57:59.441331 3203 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:57:59.441398 3203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:57:59.441937 3203 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43711 } }
I20260928 14:57:59.442020 3203 leader_election.cc:302] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c535175a1e416d80b8c7c2a36ab2e3; no voters:
I20260928 14:57:59.442214 3203 leader_election.cc:288] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:57:59.442297 3208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:57:59.442472 3208 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [term 1 LEADER]: Becoming Leader. State: Replica: d5c535175a1e416d80b8c7c2a36ab2e3, State: Running, Role: LEADER
I20260928 14:57:59.442487 3203 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:57:59.442608 3208 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43711 } }
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:59.443177 3209 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43711 } } }
I20260928 14:57:59.443341 3209 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:57:59.444162 3210 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c535175a1e416d80b8c7c2a36ab2e3. Latest consensus state: current_term: 1 leader_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c535175a1e416d80b8c7c2a36ab2e3" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43711 } } }
I20260928 14:57:59.444298 3210 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3 [sys.catalog]: This master's current role is: LEADER
W20260928 14:57:59.444706 3224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:57:59.444763 3224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:57:59.444816 3219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:57:59.444983 3219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:57:59.446943 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.435210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.1.252.254:43711)
0928 14:57:59.435383 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.435392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.436088 (+ 696us) server_negotiation.cc:407] Connection header received
0928 14:57:59.436677 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.436687 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.436748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.436902 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.437532 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.438896 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.439655 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.440365 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.440492 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.443309 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.443313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.443313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.443325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.445354 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.445747 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.445750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.445750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.445854 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.446091 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.446092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.446093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.446388 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.446540 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.446615 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.446760 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 14:57:59.447630 3219 catalog_manager.cc:1347] Generated new cluster ID: 672706c230eb42db8a879661a8adc0e0
I20260928 14:57:59.447681 3219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:57:59.456156 3219 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:57:59.456636 3219 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:57:59.460644 3219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: Generated new TSK 0
I20260928 14:57:59.460811 3219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:57:59.494967 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43711
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:59.574713 3229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:59.574903 3229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:59.574925 3229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:59.576109 3229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:59.576151 3229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:59.576171 3229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:57:59.576189 3229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:57:59.577579 3229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:59.577790 3229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:59.577991 3229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:59.582866 3234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.582882 3237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.582903 3235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:59.583165 3229 server_base.cc:1034] running on GCE node
I20260928 14:57:59.583334 3229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:59.583515 3229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:59.584649 3229 hybrid_clock.cc:648] HybridClock initialized: now 1790607479584637 us; error 28 us; skew 500 ppm
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:59.587191 3229 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:57:59.587572 3229 webserver.cc:460] Webserver started at http://127.1.252.193:33991/ using document root <none> and password file <none>
I20260928 14:57:59.587792 3229 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:59.587859 3229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:59.588020 3229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:59.589006 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "76fb1fa170e14de2bfac62be789aa557"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "60f140d055e088a5e7e4967b3a23ad4e"
server_key_iv: "2fec33112359eb77e9b04cf040e9768b"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.589368 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "76fb1fa170e14de2bfac62be789aa557"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "60f140d055e088a5e7e4967b3a23ad4e"
server_key_iv: "2fec33112359eb77e9b04cf040e9768b"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.590658 3229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:57:59.591360 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.591538 3229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:59.591634 3229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "76fb1fa170e14de2bfac62be789aa557"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "60f140d055e088a5e7e4967b3a23ad4e"
server_key_iv: "2fec33112359eb77e9b04cf040e9768b"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.591712 3229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:59.606379 3229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:59.608682 3229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:59.608850 3229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:59.609368 3229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:59.609427 3229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.609465 3229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:59.609496 3229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.616168 3229 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:43327
I20260928 14:57:59.616281 3360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:43327 every 8 connection(s)
I20260928 14:57:59.616530 3229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:59.619585 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3229
I20260928 14:57:59.619632 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.609884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:40923 (local address 127.1.252.254:43711)
0928 14:57:59.609958 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.609961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.610163 (+ 202us) server_negotiation.cc:407] Connection header received
0928 14:57:59.610658 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.610660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.610675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.610752 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.611559 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.612434 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.613988 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.614359 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.614448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.617146 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.617147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.617148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.617154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.618365 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.618766 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.618767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.618767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.618815 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.618992 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.618993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.618993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.619087 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.619138 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.619503 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.619571 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:57:59.619689 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:57:59.619853 2035 external_mini_cluster.cc:1428] Setting key 4adb6afa7fcaa28fcdcebc5110098764
I20260928 14:57:59.619875 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.609864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.1.252.193:40923)
0928 14:57:59.610035 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.610072 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.610521 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.610835 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.610840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.611186 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:59.611453 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.611459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.612549 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.612550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.613889 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.613893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.614470 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.614471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.614579 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.615120 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:59.615139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:59.617060 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:59.618449 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.618462 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.618464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.618670 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.618859 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.618860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.618860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.618945 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.619177 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:59.619183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:59.619320 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.619465 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.619634 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 14:57:59.620378 3361 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43711
I20260928 14:57:59.620460 3361 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:59.620640 3361 heartbeater.cc:511] Master 127.1.252.254:43711 requested a full tablet report, sending...
I20260928 14:57:59.622210 3156 ts_manager.cc:194] Registered new tserver with Master: 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327)
I20260928 14:57:59.622841 3156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:40923
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 14:57:59.628633 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.621371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:46725 (local address 127.1.252.254:43711)
0928 14:57:59.621404 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.621405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.623128 (+ 1723us) server_negotiation.cc:407] Connection header received
0928 14:57:59.623197 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.623198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.623208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.623251 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.623646 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.624231 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.624897 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.625101 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.625179 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.626058 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.626059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.626059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.626066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.627476 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.627822 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.627823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.627823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.627920 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.628132 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.628133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.628133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.628237 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.628325 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.628477 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.628548 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:57:59.628641 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.621315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.1.252.193:46725)
0928 14:57:59.623075 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.623086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.623142 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.623303 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.623304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.623494 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:59.623555 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.623556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.624325 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.624326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.624816 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.624817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.625176 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.625177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.625260 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.625681 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:59.625686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:59.625954 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:59.627564 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.627565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.627565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.627731 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.627975 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.627975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.627976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.628035 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.628320 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:59.628321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:59.628366 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.628452 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.628549 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1734,"thread_start_us":64,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:57:59.646627 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43711
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:59.726712 3370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:59.726958 3370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:59.726998 3370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:59.728190 3370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:59.728261 3370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:59.728295 3370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:57:59.728333 3370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:57:59.729795 3370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:59.730062 3370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:59.730296 3370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:59.735467 3376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.735503 3378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.735524 3375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:59.735707 3370 server_base.cc:1034] running on GCE node
I20260928 14:57:59.736068 3370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:59.736258 3370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:59.737396 3370 hybrid_clock.cc:648] HybridClock initialized: now 1790607479737383 us; error 28 us; skew 500 ppm
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:59.739830 3370 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:57:59.740319 3370 webserver.cc:460] Webserver started at http://127.1.252.194:46377/ using document root <none> and password file <none>
I20260928 14:57:59.740553 3370 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:59.740626 3370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:59.740774 3370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:59.741773 3370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "ca199aa574f0024e16a488dfe1978c99"
server_key_iv: "f32b2ca7621bd4c6de8ab7c482a0d4f0"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.742153 3370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "ca199aa574f0024e16a488dfe1978c99"
server_key_iv: "f32b2ca7621bd4c6de8ab7c482a0d4f0"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.743655 3370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:59.744520 3385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.744701 3370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:57:59.744796 3370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "ca199aa574f0024e16a488dfe1978c99"
server_key_iv: "f32b2ca7621bd4c6de8ab7c482a0d4f0"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.744877 3370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:59.767020 3370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:59.769340 3370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:59.769508 3370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:59.770076 3370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:59.770141 3370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.770200 3370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:59.770219 3370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:59.777563 3370 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:45081
I20260928 14:57:59.777647 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:45081 every 8 connection(s)
I20260928 14:57:59.777976 3370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:57:59.779855 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.770472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:46811 (local address 127.1.252.254:43711)
0928 14:57:59.770536 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.770538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.770732 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:57:59.771446 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.771448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.771464 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.771538 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.772378 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.773227 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.774367 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.774728 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.774812 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.776948 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.776949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.776949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.776957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.778587 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.778968 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.778969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.778970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.779020 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.779217 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.779218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.779218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.779366 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.779460 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.779733 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.779793 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:57:59.780129 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.770473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.1.252.194:46811)
0928 14:57:59.770614 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.770648 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.771290 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.771687 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.771691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.772041 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:59.772283 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.772288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.773328 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.773329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.774292 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.774295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.774836 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.774837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.774970 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.775465 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:59.775474 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:59.776872 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:59.778652 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.778656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.778658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.778893 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.779078 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.779078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.779079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.779165 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.779448 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:59.779452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:59.779595 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.779698 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.779848 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260928 14:57:59.780617 3502 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43711
I20260928 14:57:59.780695 3502 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:59.780869 3502 heartbeater.cc:511] Master 127.1.252.254:43711 requested a full tablet report, sending...
I20260928 14:57:59.781217 3155 ts_manager.cc:194] Registered new tserver with Master: d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194:45081)
I20260928 14:57:59.781219 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3370
I20260928 14:57:59.781311 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:57:59.781490 2035 external_mini_cluster.cc:1428] Setting key e033b08f5eda28643c8ea2f5cbbda6b3
I20260928 14:57:59.781672 3155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:46811
I20260928 14:57:59.787249 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.781837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:43597 (local address 127.1.252.254:43711)
0928 14:57:59.781872 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.781873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.781959 (+ 86us) server_negotiation.cc:407] Connection header received
0928 14:57:59.782019 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.782019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.782031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.782080 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.782422 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.782871 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.783567 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.783772 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.783845 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.784785 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.784786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.784787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.784793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.786056 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.786482 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.786483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.786483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.786532 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.786777 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.786778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.786778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.786902 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.786955 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.787135 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.787191 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:57:59.787369 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.781795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.1.252.194:43597)
0928 14:57:59.781909 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.781918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.781972 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.782110 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.782111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.782274 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:59.782331 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.782332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.782967 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.782968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.783481 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.783482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.783843 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.783844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.783963 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.784357 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:59.784361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:59.784652 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:59.786165 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.786166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.786166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.786394 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.786600 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.786600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.786601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.786676 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.786973 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:59.786974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:59.787028 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.787111 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.787227 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:57:59.807981 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43711
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:57:59.887153 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:57:59.887387 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:57:59.887447 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:57:59.888638 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:57:59.888679 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:57:59.888700 3511 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:57:59.888716 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:57:59.890154 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:57:59.890395 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:57:59.890621 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:57:59.896620 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.896649 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:57:59.896653 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:57:59.896839 3511 server_base.cc:1034] running on GCE node
I20260928 14:57:59.897064 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:57:59.897333 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:57:59.898478 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790607479898469 us; error 27 us; skew 500 ppm
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:57:59.900947 3511 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:57:59.901355 3511 webserver.cc:460] Webserver started at http://127.1.252.195:46593/ using document root <none> and password file <none>
I20260928 14:57:59.901588 3511 fs_manager.cc:359] Metadata directory not provided
I20260928 14:57:59.901657 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:57:59.901811 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:57:59.902848 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "54dc04586da04767b85a56b35672973d"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "987217a57105c63c27793513154d01e1"
server_key_iv: "598798f986111047510343e522e36a47"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.903200 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "54dc04586da04767b85a56b35672973d"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "987217a57105c63c27793513154d01e1"
server_key_iv: "598798f986111047510343e522e36a47"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.904795 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:57:59.905561 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.905720 3511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:57:59.905814 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "54dc04586da04767b85a56b35672973d"
format_stamp: "Formatted at 2026-09-28 14:57:59 on dist-test-slave-sf70"
server_key: "987217a57105c63c27793513154d01e1"
server_key_iv: "598798f986111047510343e522e36a47"
server_key_version: "encryptionkey@0"
I20260928 14:57:59.905900 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:57:59.916790 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:57:59.919019 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:57:59.919181 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:57:59.919657 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:57:59.919719 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:57:59.919761 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:57:59.919788 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:59.926930 3511 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:37193
I20260928 14:57:59.927060 3642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:37193 every 8 connection(s)
I20260928 14:57:59.927332 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:57:59.929589 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.920216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:56633 (local address 127.1.252.254:43711)
0928 14:57:59.920275 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.920277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.920499 (+ 222us) server_negotiation.cc:407] Connection header received
0928 14:57:59.921022 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.921023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.921036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.921118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.921973 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.922821 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.923904 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.924126 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.924215 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.926832 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.926833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.926833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.926840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.928494 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.928879 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.928880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.928880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.928932 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.929137 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.929137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.929138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.929235 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.929283 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.929471 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.929528 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:57:59.929745 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.920199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.1.252.195:56633)
0928 14:57:59.920359 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.920404 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.920922 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.921186 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.921190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.921562 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:59.921844 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.921851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.922943 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.922944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.923788 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.923791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.924238 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.924239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.924374 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.924871 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:59.924880 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:59.926719 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:59.928589 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.928594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.928596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.928803 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.928993 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.928993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.928994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.929082 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.929273 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:59.929276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:59.929374 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.929450 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.929559 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260928 14:57:59.930230 3643 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43711
I20260928 14:57:59.930313 3643 heartbeater.cc:464] Registering TS with master...
I20260928 14:57:59.930495 3643 heartbeater.cc:511] Master 127.1.252.254:43711 requested a full tablet report, sending...
I20260928 14:57:59.931120 3155 ts_manager.cc:194] Registered new tserver with Master: 54dc04586da04767b85a56b35672973d (127.1.252.195:37193)
I20260928 14:57:59.931691 3155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:56633
I20260928 14:57:59.932650 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3511
I20260928 14:57:59.932757 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:57:59.932933 2035 external_mini_cluster.cc:1428] Setting key b2583d8f5b2fec160d531f393f672bcb
I20260928 14:57:59.933758 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:57:59.937129 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.931156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.1.252.195:58675)
0928 14:57:59.931349 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.931360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.931418 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.931543 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.931544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.931744 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:57:59.931851 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.931853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.932537 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.932538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.933246 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.933248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.933646 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.933647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.933725 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.934177 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:57:59.934182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:57:59.934471 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:57:59.936198 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.936199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.936199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.936359 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.936566 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:57:59.936567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:57:59.936567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:57:59.936617 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:57:59.936890 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:57:59.936891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:57:59.936931 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.936981 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.937061 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260928 14:57:59.937216 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.931315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:58675 (local address 127.1.252.254:43711)
0928 14:57:59.931360 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.931362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.931402 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:57:59.931473 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.931474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.931484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.931527 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.931957 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.932453 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.933379 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.933564 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.933636 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.934557 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.934559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.934559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.934566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.936116 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.936436 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.936438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.936438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.936500 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.936683 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.936683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.936684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.936819 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.936898 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.937004 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.937082 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:57:59 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:57:59.956470 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.950212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.1.252.254:43711)
0928 14:57:59.950271 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.950273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.950281 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:57:59.950329 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.950330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.950338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.950392 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:57:59.950875 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.951329 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.952072 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.952248 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.952303 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.954363 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:57:59.954364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:57:59.954364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:57:59.954370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:57:59.955495 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.955788 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.955789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.955790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.955835 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:57:59.956044 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:57:59.956045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:57:59.956045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:57:59.956142 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:57:59.956220 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.956348 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.956401 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:57:59.957973 3155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:57:59.959493 3155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:57:59.965730 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.961696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.1.252.194:45081)
0928 14:57:59.961862 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.961867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.962104 (+ 237us) server_negotiation.cc:407] Connection header received
0928 14:57:59.962149 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.962151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.962173 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.962259 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:59.962443 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.963243 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.964659 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.965290 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.965377 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.965438 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.965600 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.965640 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 14:57:59.965721 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.961602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:45081 (local address 127.0.0.1:45830)
0928 14:57:59.961953 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.961964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.962025 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.962263 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.962266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.962268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:59.962331 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.962334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.963332 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.963333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.964572 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.964575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.965377 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.965378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.965468 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.965480 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.965582 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.965635 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":56,"threads_started":1}
I20260928 14:57:59.966022 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.961797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:37193 (local address 127.0.0.1:49564)
0928 14:57:59.961970 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.961980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.962025 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.962609 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.962610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.962612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:59.962653 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.962654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.963834 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.963835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.964876 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.964878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.965783 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.965784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.965861 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.965866 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.965918 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.965954 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":51,"threads_started":1}
I20260928 14:57:59.966326 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.961969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.1.252.195:37193)
0928 14:57:59.962395 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.962400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.962409 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:57:59.962457 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.962459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.962475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.962548 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:59.962737 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.963750 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.964946 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.965641 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.965698 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.965745 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.966170 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.966216 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":68,"threads_started":1}
I20260928 14:57:59.966553 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.961599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43327 (local address 127.0.0.1:41104)
0928 14:57:59.961896 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:57:59.961936 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:57:59.962012 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:57:59.962304 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:57:59.962305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:57:59.962307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:57:59.962375 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.962377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.963684 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.963685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.964776 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:57:59.964778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.966176 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.966177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:57:59.966235 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.966238 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:57:59.966292 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:57:59.966331 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":32,"thread_start_us":66,"threads_started":1}
I20260928 14:57:59.966480 3436 tablet_service.cc:1467] Processing CreateTablet for tablet b0f467300bee4c20924fd95915e7161b (DEFAULT_TABLE table=test-table [id=c0c91a60e59a438e80c3d6a215aa55f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:59.966791 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:57:59.961707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.1.252.193:43327)
0928 14:57:59.961893 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:57:59.961897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:57:59.962101 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:57:59.962144 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:57:59.962148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:57:59.962168 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:57:59.962254 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:57:59.962468 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.963595 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.964859 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:57:59.965613 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:57:59.965691 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:57:59.965745 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:57:59.966335 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 14:57:59.966383 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260928 14:57:59.966879 3436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f467300bee4c20924fd95915e7161b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:59.966921 3577 tablet_service.cc:1467] Processing CreateTablet for tablet b0f467300bee4c20924fd95915e7161b (DEFAULT_TABLE table=test-table [id=c0c91a60e59a438e80c3d6a215aa55f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:59.967253 3577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f467300bee4c20924fd95915e7161b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:59.967347 3295 tablet_service.cc:1467] Processing CreateTablet for tablet b0f467300bee4c20924fd95915e7161b (DEFAULT_TABLE table=test-table [id=c0c91a60e59a438e80c3d6a215aa55f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:57:59.967664 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f467300bee4c20924fd95915e7161b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:57:59.969942 3664 tablet_bootstrap.cc:492] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Bootstrap starting.
I20260928 14:57:59.970755 3664 tablet_bootstrap.cc:654] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:59.970788 3665 tablet_bootstrap.cc:492] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Bootstrap starting.
I20260928 14:57:59.971244 3664 log.cc:824] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:59.971289 3666 tablet_bootstrap.cc:492] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Bootstrap starting.
I20260928 14:57:59.971427 3665 tablet_bootstrap.cc:654] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:59.971757 3665 log.cc:824] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:59.971964 3666 tablet_bootstrap.cc:654] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Neither blocks nor log segments found. Creating new log.
I20260928 14:57:59.972288 3666 log.cc:824] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Log is configured to *not* fsync() on all Append() calls
I20260928 14:57:59.972517 3664 tablet_bootstrap.cc:492] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: No bootstrap required, opened a new log
I20260928 14:57:59.972594 3664 ts_tablet_manager.cc:1397] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:57:59.972934 3666 tablet_bootstrap.cc:492] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: No bootstrap required, opened a new log
I20260928 14:57:59.972986 3666 ts_tablet_manager.cc:1397] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:57:59.974035 3664 raft_consensus.cc:348] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:57:59.974196 3664 raft_consensus.cc:374] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:59.974241 3664 raft_consensus.cc:729] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dc04586da04767b85a56b35672973d, State: Initialized, Role: FOLLOWER
I20260928 14:57:59.974352 3664 consensus_queue.cc:260] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:57:59.974426 3666 raft_consensus.cc:348] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:57:59.974579 3666 raft_consensus.cc:374] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:59.974622 3666 raft_consensus.cc:729] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fb1fa170e14de2bfac62be789aa557, State: Initialized, Role: FOLLOWER
I20260928 14:57:59.974648 3664 ts_tablet_manager.cc:1428] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:57:59.974678 3643 heartbeater.cc:503] Master 127.1.252.254:43711 was elected leader, sending a full tablet report...
I20260928 14:57:59.974731 3666 consensus_queue.cc:260] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:57:59.974987 3666 ts_tablet_manager.cc:1428] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:57:59.975044 3665 tablet_bootstrap.cc:492] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: No bootstrap required, opened a new log
I20260928 14:57:59.975100 3361 heartbeater.cc:503] Master 127.1.252.254:43711 was elected leader, sending a full tablet report...
I20260928 14:57:59.975119 3665 ts_tablet_manager.cc:1397] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:57:59.976912 3665 raft_consensus.cc:348] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:57:59.977069 3665 raft_consensus.cc:374] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:57:59.977104 3665 raft_consensus.cc:729] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a8f5c6eeea49d893a0fa4c9fecb187, State: Initialized, Role: FOLLOWER
I20260928 14:57:59.977216 3665 consensus_queue.cc:260] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:57:59.977509 3665 ts_tablet_manager.cc:1428] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 14:57:59.977523 3502 heartbeater.cc:503] Master 127.1.252.254:43711 was elected leader, sending a full tablet report...
W20260928 14:58:00.028601 3503 tablet.cc:2369] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:00.039952 3672 raft_consensus.cc:482] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:00.040076 3672 raft_consensus.cc:504] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.040504 3672 leader_election.cc:288] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327), 54dc04586da04767b85a56b35672973d (127.1.252.195:37193)
I20260928 14:58:00.043728 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.040743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43327 (local address 127.1.252.194:48547)
0928 14:58:00.040876 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.040891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.040973 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.041135 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.041136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.041138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:00.041219 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.041220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.041893 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.041894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.042742 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.042744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.043394 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.043395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.043502 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.043511 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.043554 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.043616 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 14:58:00.043742 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.040976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:38071 (local address 127.1.252.195:37193)
0928 14:58:00.041028 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.041030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.041144 (+ 114us) server_negotiation.cc:407] Connection header received
0928 14:58:00.041225 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.041226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.041238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.041308 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:00.041472 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.041951 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.042947 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.043371 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.043443 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.043451 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.043640 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.043684 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:58:00.043742 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.040855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:48547 (local address 127.1.252.193:43327)
0928 14:58:00.040906 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.040908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.040965 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:58:00.041044 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.041045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.041053 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.041131 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:00.041293 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.041801 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.042849 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.043290 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.043378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.043386 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.043635 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.043680 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:58:00.043749 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.040887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:37193 (local address 127.1.252.194:38071)
0928 14:58:00.041074 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.041088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.041159 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.041318 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.041319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.041320 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:00.041400 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.041401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.042034 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.042035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.042859 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.042861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.043456 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.043457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.043519 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.043522 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.043595 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.043642 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
I20260928 14:58:00.044087 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f467300bee4c20924fd95915e7161b" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb1fa170e14de2bfac62be789aa557" is_pre_election: true
I20260928 14:58:00.044134 3597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f467300bee4c20924fd95915e7161b" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54dc04586da04767b85a56b35672973d" is_pre_election: true
I20260928 14:58:00.044239 3315 raft_consensus.cc:2454] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 0.
I20260928 14:58:00.044312 3597 raft_consensus.cc:2454] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 0.
I20260928 14:58:00.044479 3389 leader_election.cc:302] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fb1fa170e14de2bfac62be789aa557, d2a8f5c6eeea49d893a0fa4c9fecb187; no voters:
I20260928 14:58:00.044615 3672 raft_consensus.cc:2781] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:00.044670 3672 raft_consensus.cc:482] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:00.044695 3672 raft_consensus.cc:3037] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.045671 3672 raft_consensus.cc:504] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.045809 3672 leader_election.cc:288] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 election: Requested vote from peers 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327), 54dc04586da04767b85a56b35672973d (127.1.252.195:37193)
I20260928 14:58:00.045948 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f467300bee4c20924fd95915e7161b" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb1fa170e14de2bfac62be789aa557"
I20260928 14:58:00.046029 3315 raft_consensus.cc:3037] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.045994 3597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f467300bee4c20924fd95915e7161b" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54dc04586da04767b85a56b35672973d"
I20260928 14:58:00.046077 3597 raft_consensus.cc:3037] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.046712 3315 raft_consensus.cc:2454] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 1.
I20260928 14:58:00.046718 3597 raft_consensus.cc:2454] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 1.
I20260928 14:58:00.046896 3389 leader_election.cc:302] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54dc04586da04767b85a56b35672973d, d2a8f5c6eeea49d893a0fa4c9fecb187; no voters:
I20260928 14:58:00.047000 3672 raft_consensus.cc:2781] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:00.047143 3672 raft_consensus.cc:686] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 LEADER]: Becoming Leader. State: Replica: d2a8f5c6eeea49d893a0fa4c9fecb187, State: Running, Role: LEADER
I20260928 14:58:00.047215 3672 consensus_queue.cc:237] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.048074 3155 catalog_manager.cc:5495] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:00.060276 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.057984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.1.252.194:45081)
0928 14:58:00.058039 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.058041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.058066 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:58:00.058140 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.058141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.058156 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.058212 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:00.058323 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.058761 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.059504 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.059681 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.059731 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.059945 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.060038 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.060094 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.060149 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.060167 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:58:00.061882 3597 raft_consensus.cc:1261] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Refusing update from remote peer d2a8f5c6eeea49d893a0fa4c9fecb187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:00.061947 3315 raft_consensus.cc:1261] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Refusing update from remote peer d2a8f5c6eeea49d893a0fa4c9fecb187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:00.062219 3672 consensus_queue.cc:1035] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:00.062355 3674 consensus_queue.cc:1035] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:00.064637 3679 mvcc.cc:205] Tried to move back new op lower bound from 7334328238331043840 to 7334328238274048000. Current Snapshot: MvccSnapshot[applied={T|T < 7334328238331043840}]
I20260928 14:58:00.067786 3155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:00.067879 3155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:00.068676 3155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: Sending DeleteTablet for 3 replicas of tablet b0f467300bee4c20924fd95915e7161b
I20260928 14:58:00.069036 3436 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f467300bee4c20924fd95915e7161b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:00 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:45830
I20260928 14:58:00.069028 3295 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f467300bee4c20924fd95915e7161b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:00 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:41104
I20260928 14:58:00.069264 3686 tablet_replica.cc:330] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: stopping tablet replica
I20260928 14:58:00.069367 3686 raft_consensus.cc:2229] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:00.069375 3577 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f467300bee4c20924fd95915e7161b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:00 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:49564
I20260928 14:58:00.069326 3155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:58:00.069478 3686 raft_consensus.cc:2258] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:58:00.069482 3155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:00.069626 3687 tablet_replica.cc:330] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: stopping tablet replica
I20260928 14:58:00.069712 3687 raft_consensus.cc:2229] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:00.069813 3687 raft_consensus.cc:2258] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:00.070297 3686 ts_tablet_manager.cc:1905] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:00.070734 3436 tablet_service.cc:1467] Processing CreateTablet for tablet 55754c38209d4410873a20105fc97574 (DEFAULT_TABLE table=test-table [id=9656610e09f248cdaaf85cf9cec64a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:00.070855 3436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55754c38209d4410873a20105fc97574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:00.070963 3295 tablet_service.cc:1467] Processing CreateTablet for tablet 55754c38209d4410873a20105fc97574 (DEFAULT_TABLE table=test-table [id=9656610e09f248cdaaf85cf9cec64a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:00.071084 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55754c38209d4410873a20105fc97574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:00.072331 3665 tablet_bootstrap.cc:492] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Bootstrap starting.
I20260928 14:58:00.072368 3686 ts_tablet_manager.cc:1918] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:00.072528 3686 log.cc:1196] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/b0f467300bee4c20924fd95915e7161b
I20260928 14:58:00.072695 3577 tablet_service.cc:1467] Processing CreateTablet for tablet 55754c38209d4410873a20105fc97574 (DEFAULT_TABLE table=test-table [id=9656610e09f248cdaaf85cf9cec64a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:00.072769 3665 tablet_bootstrap.cc:654] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:00.072825 3577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55754c38209d4410873a20105fc97574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:00.073032 3686 ts_tablet_manager.cc:1939] T b0f467300bee4c20924fd95915e7161b P 76fb1fa170e14de2bfac62be789aa557: Deleting consensus metadata
I20260928 14:58:00.073122 3666 tablet_bootstrap.cc:492] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Bootstrap starting.
I20260928 14:58:00.073518 3666 tablet_bootstrap.cc:654] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:00.073797 3688 tablet_replica.cc:330] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: stopping tablet replica
I20260928 14:58:00.074043 3688 raft_consensus.cc:2229] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:00.074128 3664 tablet_bootstrap.cc:492] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Bootstrap starting.
I20260928 14:58:00.074265 3688 raft_consensus.cc:2258] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:00.074565 3664 tablet_bootstrap.cc:654] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:00.074702 3666 tablet_bootstrap.cc:492] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: No bootstrap required, opened a new log
I20260928 14:58:00.074702 3665 tablet_bootstrap.cc:492] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: No bootstrap required, opened a new log
I20260928 14:58:00.074743 3666 ts_tablet_manager.cc:1397] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:58:00.074774 3665 ts_tablet_manager.cc:1397] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 14:58:00.074909 3666 raft_consensus.cc:348] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.074985 3666 raft_consensus.cc:374] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:00.074937 3665 raft_consensus.cc:348] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.075011 3666 raft_consensus.cc:729] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fb1fa170e14de2bfac62be789aa557, State: Initialized, Role: FOLLOWER
I20260928 14:58:00.075022 3665 raft_consensus.cc:374] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:00.075047 3665 raft_consensus.cc:729] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a8f5c6eeea49d893a0fa4c9fecb187, State: Initialized, Role: FOLLOWER
I20260928 14:58:00.075067 3666 consensus_queue.cc:260] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.075078 3144 catalog_manager.cc:4841] TS 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327): tablet b0f467300bee4c20924fd95915e7161b (table test-table [id=c0c91a60e59a438e80c3d6a215aa55f8]) successfully deleted
I20260928 14:58:00.075093 3665 consensus_queue.cc:260] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.075165 3666 ts_tablet_manager.cc:1428] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:00.075181 3665 ts_tablet_manager.cc:1428] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:00.075196 3664 tablet_bootstrap.cc:492] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: No bootstrap required, opened a new log
I20260928 14:58:00.075239 3664 ts_tablet_manager.cc:1397] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:00.074695 3687 ts_tablet_manager.cc:1905] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:00.075392 3664 raft_consensus.cc:348] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.075449 3664 raft_consensus.cc:374] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:00.075465 3664 raft_consensus.cc:729] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dc04586da04767b85a56b35672973d, State: Initialized, Role: FOLLOWER
I20260928 14:58:00.075502 3664 consensus_queue.cc:260] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.075591 3664 ts_tablet_manager.cc:1428] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:00.076135 3688 ts_tablet_manager.cc:1905] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:00.076836 3687 ts_tablet_manager.cc:1918] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:00.076915 3687 log.cc:1196] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/b0f467300bee4c20924fd95915e7161b
I20260928 14:58:00.077173 3687 ts_tablet_manager.cc:1939] T b0f467300bee4c20924fd95915e7161b P d2a8f5c6eeea49d893a0fa4c9fecb187: Deleting consensus metadata
I20260928 14:58:00.077617 3141 catalog_manager.cc:4841] TS d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194:45081): tablet b0f467300bee4c20924fd95915e7161b (table test-table [id=c0c91a60e59a438e80c3d6a215aa55f8]) successfully deleted
I20260928 14:58:00.077782 3688 ts_tablet_manager.cc:1918] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:00.077870 3688 log.cc:1196] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/b0f467300bee4c20924fd95915e7161b
I20260928 14:58:00.078112 3688 ts_tablet_manager.cc:1939] T b0f467300bee4c20924fd95915e7161b P 54dc04586da04767b85a56b35672973d: Deleting consensus metadata
I20260928 14:58:00.078518 3144 catalog_manager.cc:4841] TS 54dc04586da04767b85a56b35672973d (127.1.252.195:37193): tablet b0f467300bee4c20924fd95915e7161b (table test-table [id=c0c91a60e59a438e80c3d6a215aa55f8]) successfully deleted
W20260928 14:58:00.117302 3362 tablet.cc:2369] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:58:00.177948 3644 tablet.cc:2369] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:00.224159 3672 raft_consensus.cc:482] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:00.224382 3672 raft_consensus.cc:504] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.224579 3672 leader_election.cc:288] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327), 54dc04586da04767b85a56b35672973d (127.1.252.195:37193)
I20260928 14:58:00.224920 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55754c38209d4410873a20105fc97574" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb1fa170e14de2bfac62be789aa557" is_pre_election: true
I20260928 14:58:00.224920 3597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55754c38209d4410873a20105fc97574" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54dc04586da04767b85a56b35672973d" is_pre_election: true
I20260928 14:58:00.225034 3597 raft_consensus.cc:2454] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 0.
I20260928 14:58:00.225034 3315 raft_consensus.cc:2454] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 0.
I20260928 14:58:00.225194 3389 leader_election.cc:302] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54dc04586da04767b85a56b35672973d, d2a8f5c6eeea49d893a0fa4c9fecb187; no voters:
I20260928 14:58:00.225301 3672 raft_consensus.cc:2781] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:00.225340 3672 raft_consensus.cc:482] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:00.225366 3672 raft_consensus.cc:3037] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.226014 3672 raft_consensus.cc:504] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.226136 3672 leader_election.cc:288] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 election: Requested vote from peers 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327), 54dc04586da04767b85a56b35672973d (127.1.252.195:37193)
I20260928 14:58:00.226303 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55754c38209d4410873a20105fc97574" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb1fa170e14de2bfac62be789aa557"
I20260928 14:58:00.226347 3597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55754c38209d4410873a20105fc97574" candidate_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54dc04586da04767b85a56b35672973d"
I20260928 14:58:00.226409 3597 raft_consensus.cc:3037] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.226411 3315 raft_consensus.cc:3037] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.227020 3597 raft_consensus.cc:2454] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 1.
I20260928 14:58:00.227027 3315 raft_consensus.cc:2454] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a8f5c6eeea49d893a0fa4c9fecb187 in term 1.
I20260928 14:58:00.227198 3389 leader_election.cc:302] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54dc04586da04767b85a56b35672973d, d2a8f5c6eeea49d893a0fa4c9fecb187; no voters:
I20260928 14:58:00.227306 3672 raft_consensus.cc:2781] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:00.227363 3672 raft_consensus.cc:686] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 LEADER]: Becoming Leader. State: Replica: d2a8f5c6eeea49d893a0fa4c9fecb187, State: Running, Role: LEADER
I20260928 14:58:00.227414 3672 consensus_queue.cc:237] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.228026 3155 catalog_manager.cc:5495] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:58:00 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:00.288760 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.280828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43711 (local address 127.0.0.1:35022)
0928 14:58:00.281039 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.281048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.281132 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.281310 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.281311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.282207 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:00.282262 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.282265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.282955 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.282956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.283483 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.283484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.283839 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.283840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.283927 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.284443 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:00.284448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:00.285932 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:00.287602 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:00.287611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:00.287614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:00.287836 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:00.288095 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:00.288096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:00.288097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:00.288160 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:00.288442 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:00.288449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:00.288495 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.288564 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.288661 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":121,"threads_started":1}
I20260928 14:58:00.288830 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.280899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.1.252.254:43711)
0928 14:58:00.280980 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.280983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.281112 (+ 129us) server_negotiation.cc:407] Connection header received
0928 14:58:00.281223 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.281224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.281240 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.281323 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:00.282354 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.282870 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.283572 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.283756 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.283827 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.286032 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:00.286033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:00.286033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:00.286040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:00.287512 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:00.287955 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:00.287957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:00.287957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:00.288015 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:00.288243 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:00.288244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:00.288245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:00.288360 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:00.288419 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.288598 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.288709 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":33}
I20260928 14:58:00.290855 3155 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:35022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:58:00.291038 3155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:00.292541 3436 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe1d342fb5845819ae256bf8ff865eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc52189542484d299aee576b367a4852]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:00.292553 3577 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe1d342fb5845819ae256bf8ff865eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc52189542484d299aee576b367a4852]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:00.292707 3577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe1d342fb5845819ae256bf8ff865eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:00.292707 3436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe1d342fb5845819ae256bf8ff865eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:00.292688 3295 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe1d342fb5845819ae256bf8ff865eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc52189542484d299aee576b367a4852]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:00.292836 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe1d342fb5845819ae256bf8ff865eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:00.294194 3666 tablet_bootstrap.cc:492] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: Bootstrap starting.
I20260928 14:58:00.294273 3665 tablet_bootstrap.cc:492] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: Bootstrap starting.
I20260928 14:58:00.294354 3664 tablet_bootstrap.cc:492] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: Bootstrap starting.
I20260928 14:58:00.294621 3665 tablet_bootstrap.cc:654] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:00.294624 3666 tablet_bootstrap.cc:654] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:00.294788 3664 tablet_bootstrap.cc:654] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:00.295176 3665 tablet_bootstrap.cc:492] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: No bootstrap required, opened a new log
I20260928 14:58:00.295222 3665 ts_tablet_manager.cc:1397] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:58:00.295356 3665 raft_consensus.cc:348] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.295408 3665 raft_consensus.cc:374] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:00.295421 3665 raft_consensus.cc:729] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a8f5c6eeea49d893a0fa4c9fecb187, State: Initialized, Role: FOLLOWER
I20260928 14:58:00.295439 3664 tablet_bootstrap.cc:492] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: No bootstrap required, opened a new log
I20260928 14:58:00.295478 3664 ts_tablet_manager.cc:1397] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:00.295480 3665 consensus_queue.cc:260] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.295573 3665 ts_tablet_manager.cc:1428] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:00.295656 3664 raft_consensus.cc:348] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.295655 3672 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.295735 3664 raft_consensus.cc:374] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:00.295740 3672 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.295760 3664 raft_consensus.cc:729] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dc04586da04767b85a56b35672973d, State: Initialized, Role: FOLLOWER
I20260928 14:58:00.295816 3664 consensus_queue.cc:260] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.295925 3664 ts_tablet_manager.cc:1428] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:00.296007 3670 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.296128 3670 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.296186 3666 tablet_bootstrap.cc:492] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: No bootstrap required, opened a new log
I20260928 14:58:00.296242 3666 ts_tablet_manager.cc:1397] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:58:00.296394 3666 raft_consensus.cc:348] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.296465 3666 raft_consensus.cc:374] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:00.296490 3666 raft_consensus.cc:729] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fb1fa170e14de2bfac62be789aa557, State: Initialized, Role: FOLLOWER
I20260928 14:58:00.296535 3666 consensus_queue.cc:260] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.296640 3666 ts_tablet_manager.cc:1428] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:00.296746 3671 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.296841 3671 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.393383 3670 raft_consensus.cc:482] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:00.393474 3670 raft_consensus.cc:504] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.393848 3670 leader_election.cc:288] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327), d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194:45081)
I20260928 14:58:00.396821 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.394130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:43325 (local address 127.1.252.194:45081)
0928 14:58:00.394181 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.394183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.394191 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:00.394222 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.394227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.394235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.394305 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:00.394485 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.395027 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.396090 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.396454 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.396552 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.396560 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.396721 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.396756 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:58:00.396821 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.394019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:45081 (local address 127.1.252.195:43325)
0928 14:58:00.394090 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.394102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.394146 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.394335 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.394336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.394338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:00.394407 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.394410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.395109 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.395109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.395994 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.395996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.396550 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.396551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.396635 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.396638 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.396711 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.396749 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:58:00.397130 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.394024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43327 (local address 127.1.252.195:40187)
0928 14:58:00.394230 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.394238 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.394312 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.394465 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.394466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.394468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:00.394550 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.394551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.395173 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.395174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.396315 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.396317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.396864 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.396865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.396943 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.396947 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.397003 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.397044 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260928 14:58:00.397123 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.394086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:40187 (local address 127.1.252.193:43327)
0928 14:58:00.394160 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.394163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.394298 (+ 135us) server_negotiation.cc:407] Connection header received
0928 14:58:00.394367 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.394369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.394378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.394455 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:00.394631 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.395089 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.396403 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.396758 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.396844 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.396853 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.397023 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.397061 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:58:00.397112 3456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe1d342fb5845819ae256bf8ff865eb" candidate_uuid: "54dc04586da04767b85a56b35672973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" is_pre_election: true
I20260928 14:58:00.397238 3456 raft_consensus.cc:2454] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54dc04586da04767b85a56b35672973d in term 0.
I20260928 14:58:00.397459 3527 leader_election.cc:302] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54dc04586da04767b85a56b35672973d, d2a8f5c6eeea49d893a0fa4c9fecb187; no voters:
I20260928 14:58:00.397604 3670 raft_consensus.cc:2781] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:00.397574 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe1d342fb5845819ae256bf8ff865eb" candidate_uuid: "54dc04586da04767b85a56b35672973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb1fa170e14de2bfac62be789aa557" is_pre_election: true
I20260928 14:58:00.397656 3670 raft_consensus.cc:482] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:00.397681 3315 raft_consensus.cc:2454] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54dc04586da04767b85a56b35672973d in term 0.
I20260928 14:58:00.397676 3670 raft_consensus.cc:3037] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.398370 3670 raft_consensus.cc:504] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.398521 3670 leader_election.cc:288] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [CANDIDATE]: Term 1 election: Requested vote from peers 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327), d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194:45081)
I20260928 14:58:00.398645 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe1d342fb5845819ae256bf8ff865eb" candidate_uuid: "54dc04586da04767b85a56b35672973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb1fa170e14de2bfac62be789aa557"
I20260928 14:58:00.398649 3456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe1d342fb5845819ae256bf8ff865eb" candidate_uuid: "54dc04586da04767b85a56b35672973d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187"
I20260928 14:58:00.398718 3315 raft_consensus.cc:3037] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.398718 3456 raft_consensus.cc:3037] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:00.399447 3315 raft_consensus.cc:2454] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54dc04586da04767b85a56b35672973d in term 1.
I20260928 14:58:00.399483 3456 raft_consensus.cc:2454] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54dc04586da04767b85a56b35672973d in term 1.
I20260928 14:58:00.399619 3530 leader_election.cc:302] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54dc04586da04767b85a56b35672973d, 76fb1fa170e14de2bfac62be789aa557; no voters:
I20260928 14:58:00.399755 3670 raft_consensus.cc:2781] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:00.399935 3670 raft_consensus.cc:686] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [term 1 LEADER]: Becoming Leader. State: Replica: 54dc04586da04767b85a56b35672973d, State: Running, Role: LEADER
I20260928 14:58:00.399996 3670 consensus_queue.cc:237] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } }
I20260928 14:58:00.400347 3703 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: TxnStatusTablet state changed. Reason: New leader 54dc04586da04767b85a56b35672973d. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.400498 3703 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:00.400707 3705 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:58:00.400713 3155 catalog_manager.cc:5495] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d reported cstate change: term changed from 0 to 1, leader changed from <none> to 54dc04586da04767b85a56b35672973d (127.1.252.195). New cstate: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } health_report { overall_health: HEALTHY } } }
I20260928 14:58:00.400823 3705 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:58:00.413697 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.411266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.1.252.195:37193)
0928 14:58:00.411323 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.411325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.411352 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:58:00.411443 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.411444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.411462 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.411537 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:00.411670 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.412197 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.412946 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.413126 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.413201 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.413393 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.413488 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.413564 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.413618 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.413636 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:58:00.413738 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.411213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:37193 (local address 127.0.0.1:49572)
0928 14:58:00.411308 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.411319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.411373 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.411545 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.411545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.411547 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:00.411594 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.411596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.412278 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.412280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.412841 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.412843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.413201 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.413202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.413270 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.413291 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.413561 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.413571 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.413616 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.413662 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:58:00.414877 3315 raft_consensus.cc:1261] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Refusing update from remote peer 54dc04586da04767b85a56b35672973d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:00.414991 3456 raft_consensus.cc:1261] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Refusing update from remote peer 54dc04586da04767b85a56b35672973d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:00.415102 3703 consensus_queue.cc:1035] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:00.415340 3670 consensus_queue.cc:1035] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:00.415727 3671 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: TxnStatusTablet state changed. Reason: New leader 54dc04586da04767b85a56b35672973d. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.415825 3671 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.416108 3672 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: TxnStatusTablet state changed. Reason: New leader 54dc04586da04767b85a56b35672973d. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.416199 3672 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.416668 3703 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.416743 3703 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:00.416785 3703 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.416829 3703 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 54dc04586da04767b85a56b35672973d: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:00.416843 3678 mvcc.cc:205] Tried to move back new op lower bound from 7334328239777447936 to 7334328239718866944. Current Snapshot: MvccSnapshot[applied={T|T < 7334328239777447936}]
I20260928 14:58:00.416934 3671 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.416996 3671 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P 76fb1fa170e14de2bfac62be789aa557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.417424 3677 mvcc.cc:205] Tried to move back new op lower bound from 7334328239777447936 to 7334328239718866944. Current Snapshot: MvccSnapshot[applied={T|T < 7334328239777447936}]
I20260928 14:58:00.417598 3672 tablet_replica.cc:439] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54dc04586da04767b85a56b35672973d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 } } peers { permanent_uuid: "d2a8f5c6eeea49d893a0fa4c9fecb187" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 45081 } } peers { permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 } } }
I20260928 14:58:00.417695 3672 tablet_replica.cc:442] T 4fe1d342fb5845819ae256bf8ff865eb P d2a8f5c6eeea49d893a0fa4c9fecb187: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:00.423141 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.420301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:35581 (local address 127.1.252.195:37193)
0928 14:58:00.420336 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.420337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.420504 (+ 167us) server_negotiation.cc:407] Connection header received
0928 14:58:00.420567 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.420568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.420578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.420630 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:00.420798 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.421278 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.422380 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.422592 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.422656 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.422858 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.422887 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.422950 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.423074 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.423095 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:00.423168 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.420262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:37193 (local address 127.1.252.194:35581)
0928 14:58:00.420440 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.420449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.420520 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.420656 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.420658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.420660 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:00.420727 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.420728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.421358 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.421360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.422293 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.422295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.422677 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.422678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.422740 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.422754 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.422977 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.422979 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.423029 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.423080 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260928 14:58:00.425326 3597 raft_consensus.cc:1261] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Refusing update from remote peer d2a8f5c6eeea49d893a0fa4c9fecb187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:00.425367 3315 raft_consensus.cc:1261] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Refusing update from remote peer d2a8f5c6eeea49d893a0fa4c9fecb187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:00.425551 3672 consensus_queue.cc:1035] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dc04586da04767b85a56b35672973d" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:00.425681 3672 consensus_queue.cc:1035] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fb1fa170e14de2bfac62be789aa557" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:00.434312 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.431451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:58811 (local address 127.1.252.194:45081)
0928 14:58:00.431491 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.431493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.431502 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:00.431570 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.431571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.431581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.431645 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:00.431741 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.432516 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.433524 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.433828 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.433900 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.434081 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.434116 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.434169 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.434227 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.434247 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:00.434335 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.431394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:45081 (local address 127.1.252.195:58811)
0928 14:58:00.431453 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:00.431465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:00.431504 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:00.431646 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:00.431647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:00.431648 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:00.431698 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.431699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.432595 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.432596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.433435 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:00.433438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.433911 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.433912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:00.433980 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.433995 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:00.434179 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:00.434181 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:00.434225 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:58:00.434267 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 14:58:00.442339 3155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:00.442461 3155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:00.443130 3155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5c535175a1e416d80b8c7c2a36ab2e3: Sending DeleteTablet for 3 replicas of tablet 55754c38209d4410873a20105fc97574
I20260928 14:58:00.443367 3295 tablet_service.cc:1514] Processing DeleteTablet for tablet 55754c38209d4410873a20105fc97574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:00 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:41104
I20260928 14:58:00.443399 3436 tablet_service.cc:1514] Processing DeleteTablet for tablet 55754c38209d4410873a20105fc97574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:00 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:45830
I20260928 14:58:00.443464 3686 tablet_replica.cc:330] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: stopping tablet replica
I20260928 14:58:00.443501 3687 tablet_replica.cc:330] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: stopping tablet replica
I20260928 14:58:00.443512 3686 raft_consensus.cc:2229] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:00.443550 3686 raft_consensus.cc:2258] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:00.443562 3687 raft_consensus.cc:2229] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:00.443620 3687 raft_consensus.cc:2258] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:00.443867 3686 ts_tablet_manager.cc:1905] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:00.443964 3687 ts_tablet_manager.cc:1905] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:00.444133 3577 tablet_service.cc:1514] Processing DeleteTablet for tablet 55754c38209d4410873a20105fc97574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:00 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:49564
I20260928 14:58:00.444214 3688 tablet_replica.cc:330] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: stopping tablet replica
I20260928 14:58:00.444252 3688 raft_consensus.cc:2229] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:00.444278 3688 raft_consensus.cc:2258] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:00.444530 3688 ts_tablet_manager.cc:1905] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:00.445590 3686 ts_tablet_manager.cc:1918] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:00.445664 3686 log.cc:1196] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/55754c38209d4410873a20105fc97574
I20260928 14:58:00.445884 3687 ts_tablet_manager.cc:1918] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:00.445909 3686 ts_tablet_manager.cc:1939] T 55754c38209d4410873a20105fc97574 P 76fb1fa170e14de2bfac62be789aa557: Deleting consensus metadata
I20260928 14:58:00.445940 3687 log.cc:1196] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/55754c38209d4410873a20105fc97574
I20260928 14:58:00.446126 3688 ts_tablet_manager.cc:1918] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:00.446154 3687 ts_tablet_manager.cc:1939] T 55754c38209d4410873a20105fc97574 P d2a8f5c6eeea49d893a0fa4c9fecb187: Deleting consensus metadata
I20260928 14:58:00.446175 3688 log.cc:1196] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/55754c38209d4410873a20105fc97574
I20260928 14:58:00.446205 3144 catalog_manager.cc:4841] TS 76fb1fa170e14de2bfac62be789aa557 (127.1.252.193:43327): tablet 55754c38209d4410873a20105fc97574 (table test-table [id=9656610e09f248cdaaf85cf9cec64a8b]) successfully deleted
I20260928 14:58:00.446370 3688 ts_tablet_manager.cc:1939] T 55754c38209d4410873a20105fc97574 P 54dc04586da04767b85a56b35672973d: Deleting consensus metadata
I20260928 14:58:00.446461 3141 catalog_manager.cc:4841] TS d2a8f5c6eeea49d893a0fa4c9fecb187 (127.1.252.194:45081): tablet 55754c38209d4410873a20105fc97574 (table test-table [id=9656610e09f248cdaaf85cf9cec64a8b]) successfully deleted
I20260928 14:58:00.446866 3144 catalog_manager.cc:4841] TS 54dc04586da04767b85a56b35672973d (127.1.252.195:37193): tablet 55754c38209d4410873a20105fc97574 (table test-table [id=9656610e09f248cdaaf85cf9cec64a8b]) successfully deleted
Sep 28 14:58:00 dist-test-slave-sf70 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607479, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:58:00.455297 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.444281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.1.252.193:43327)
0928 14:58:00.445638 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.445640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.446709 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 14:58:00.446819 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.446820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.446833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.446904 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:00.447609 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.448156 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.449011 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.449196 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.449266 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.452385 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:00.452394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:00.452395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:00.452403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:00.454099 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:00.454489 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:00.454491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:00.454491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:00.454565 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:00.454780 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:00.454781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:00.454782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:00.454923 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:00.455077 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.455126 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.455201 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333}
W20260928 14:58:00.455560 3355 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41116
I20260928 14:58:00.462137 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:00.456511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.1.252.254:43711)
0928 14:58:00.456559 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:00.456561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:00.456569 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:00.456640 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:00.456641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:00.456651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:00.456698 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:00.457169 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.457642 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.458314 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:00.458527 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:00.458602 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:00.459686 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:00.459687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:00.459688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:00.459695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:00.461117 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:00.461498 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:00.461498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:00.461499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:00.461546 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:00.461736 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:00.461737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:00.461737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:00.461828 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:00.461898 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:00.462029 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:58:00.462082 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 14:58:00.462581 3155 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35026
I20260928 14:58:00.463131 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3229
I20260928 14:58:00.470666 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3370
I20260928 14:58:00.476732 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3511
I20260928 14:58:00.482263 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3125
2026-09-28T14:58:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2331 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:00 dist-test-slave-sf70 krb5kdc[3725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:00 dist-test-slave-sf70 krb5kdc[3725](info): set up 2 sockets
Sep 28 14:58:00 dist-test-slave-sf70 krb5kdc[3725](info): commencing operation
krb5kdc: starting...
W20260928 14:58:01.517241 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:01 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:01.635079 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41375
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:01.714789 3738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:01.715025 3738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:01.715065 3738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:01.716225 3738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:01.716292 3738 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:58:01.716318 3738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:01.716351 3738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:01.716375 3738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:01.716403 3738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:01.717866 3738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41375
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:01.718143 3738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:01.718364 3738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:01.723209 3746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:01.723232 3743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:01.723264 3744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:01.723495 3738 server_base.cc:1034] running on GCE node
I20260928 14:58:01.723783 3738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:01.724107 3738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:01.725286 3738 hybrid_clock.cc:648] HybridClock initialized: now 1790607481725268 us; error 26 us; skew 500 ppm
Sep 28 14:58:01 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:01.727713 3738 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:01.728205 3738 webserver.cc:460] Webserver started at http://127.1.252.254:44467/ using document root <none> and password file <none>
I20260928 14:58:01.728446 3738 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:01.728518 3738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:01.728667 3738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:01.729741 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "d085be63bff847e4bbe45145e90c7918"
format_stamp: "Formatted at 2026-09-28 14:58:01 on dist-test-slave-sf70"
server_key: "b1611bc5bfe7e10a17d601be88577e2c"
server_key_iv: "aab76cf650c17626d37d2ad3057a7328"
server_key_version: "encryptionkey@0"
I20260928 14:58:01.730110 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "d085be63bff847e4bbe45145e90c7918"
format_stamp: "Formatted at 2026-09-28 14:58:01 on dist-test-slave-sf70"
server_key: "b1611bc5bfe7e10a17d601be88577e2c"
server_key_iv: "aab76cf650c17626d37d2ad3057a7328"
server_key_version: "encryptionkey@0"
I20260928 14:58:01.731418 3738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:01.732165 3753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:01.732340 3738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:01.732440 3738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "d085be63bff847e4bbe45145e90c7918"
format_stamp: "Formatted at 2026-09-28 14:58:01 on dist-test-slave-sf70"
server_key: "b1611bc5bfe7e10a17d601be88577e2c"
server_key_iv: "aab76cf650c17626d37d2ad3057a7328"
server_key_version: "encryptionkey@0"
I20260928 14:58:01.732525 3738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:01.743769 3738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:01.745611 3738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:01.745785 3738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:01.750373 3738 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:41375
I20260928 14:58:01.750455 3816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:41375 every 8 connection(s)
I20260928 14:58:01.750788 3738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:01.751480 3817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:01.753756 3817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Bootstrap starting.
I20260928 14:58:01.754336 3817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:01.754637 3817 log.cc:824] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:01.755599 3817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: No bootstrap required, opened a new log
I20260928 14:58:01.757723 3817 raft_consensus.cc:348] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085be63bff847e4bbe45145e90c7918" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41375 } }
I20260928 14:58:01.757858 3817 raft_consensus.cc:374] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:01.757880 3817 raft_consensus.cc:729] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d085be63bff847e4bbe45145e90c7918, State: Initialized, Role: FOLLOWER
I20260928 14:58:01.758049 3817 consensus_queue.cc:260] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085be63bff847e4bbe45145e90c7918" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41375 } }
I20260928 14:58:01.758112 3817 raft_consensus.cc:388] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:01.758133 3817 raft_consensus.cc:482] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:01.758194 3817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:01.758708 3817 raft_consensus.cc:504] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085be63bff847e4bbe45145e90c7918" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41375 } }
I20260928 14:58:01.758791 3817 leader_election.cc:302] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d085be63bff847e4bbe45145e90c7918; no voters:
I20260928 14:58:01.758965 3817 leader_election.cc:288] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:01.759011 3820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:01.759192 3820 raft_consensus.cc:686] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [term 1 LEADER]: Becoming Leader. State: Replica: d085be63bff847e4bbe45145e90c7918, State: Running, Role: LEADER
I20260928 14:58:01.759251 3817 sys_catalog.cc:564] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:01.759373 3820 consensus_queue.cc:237] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085be63bff847e4bbe45145e90c7918" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41375 } }
I20260928 14:58:01.759537 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3738
I20260928 14:58:01.759629 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:01.759824 2035 external_mini_cluster.cc:1428] Setting key 9b4b31ef95cdcb203dfc2b94a27d5406
I20260928 14:58:01.760525 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d085be63bff847e4bbe45145e90c7918. Latest consensus state: current_term: 1 leader_uuid: "d085be63bff847e4bbe45145e90c7918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085be63bff847e4bbe45145e90c7918" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41375 } } }
I20260928 14:58:01.760533 3821 sys_catalog.cc:455] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d085be63bff847e4bbe45145e90c7918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085be63bff847e4bbe45145e90c7918" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41375 } } }
I20260928 14:58:01.760743 3821 sys_catalog.cc:458] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:01.760843 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918 [sys.catalog]: This master's current role is: LEADER
W20260928 14:58:01.761277 3836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:58:01.761384 3836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:58:01.761541 3838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:01.761720 3838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:01.763200 3838 catalog_manager.cc:1347] Generated new cluster ID: 681ad3b221984b48a3b4ac417e0edcf3
I20260928 14:58:01.763242 3838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:01 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:01.771656 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:01.760704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.1.252.254:41375)
0928 14:58:01.760961 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:01.760970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:01.761005 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:58:01.761127 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:01.761135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:01.761193 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:01.761358 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:01.762098 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.763449 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.764558 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.765131 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.765227 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:01.768041 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:01.768045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:01.768045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:01.768059 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:01.770024 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:01.770415 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:01.770418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:01.770419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:01.770543 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:01.770744 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:01.770744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:01.770745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:01.771071 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:01.771229 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:01.771309 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:58:01.771448 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260928 14:58:01.800729 3838 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:01.801272 3838 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:01.805572 3838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Generated new TSK 0
I20260928 14:58:01.805816 3838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:01.836555 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41375
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:01.923437 3843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:01.923642 3843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:01.923667 3843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:01.924896 3843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:01.924939 3843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:01.924961 3843 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:58:01.925007 3843 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:58:01.925036 3843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:01.926520 3843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:01.926779 3843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:01.927011 3843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:58:01.932142 3843 server_base.cc:1034] running on GCE node
W20260928 14:58:01.932036 3849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:01.932044 3851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:01.932036 3848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:01.932508 3843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:01.932703 3843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:01.933838 3843 hybrid_clock.cc:648] HybridClock initialized: now 1790607481933825 us; error 27 us; skew 500 ppm
Sep 28 14:58:01 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:01.936682 3843 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:01.937129 3843 webserver.cc:460] Webserver started at http://127.1.252.193:36053/ using document root <none> and password file <none>
I20260928 14:58:01.937361 3843 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:01.937413 3843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:01.937589 3843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:01.938651 3843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "7c76786d2a924821a1b621a8c4d60605"
format_stamp: "Formatted at 2026-09-28 14:58:01 on dist-test-slave-sf70"
server_key: "49e40cc7224a82755f465dc3ea259662"
server_key_iv: "a3951f17b92d0ee2682fcbb69e6bc5ad"
server_key_version: "encryptionkey@0"
I20260928 14:58:01.939011 3843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "7c76786d2a924821a1b621a8c4d60605"
format_stamp: "Formatted at 2026-09-28 14:58:01 on dist-test-slave-sf70"
server_key: "49e40cc7224a82755f465dc3ea259662"
server_key_iv: "a3951f17b92d0ee2682fcbb69e6bc5ad"
server_key_version: "encryptionkey@0"
I20260928 14:58:01.940419 3843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:01.941223 3858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:01.941401 3843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:01.941516 3843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "7c76786d2a924821a1b621a8c4d60605"
format_stamp: "Formatted at 2026-09-28 14:58:01 on dist-test-slave-sf70"
server_key: "49e40cc7224a82755f465dc3ea259662"
server_key_iv: "a3951f17b92d0ee2682fcbb69e6bc5ad"
server_key_version: "encryptionkey@0"
I20260928 14:58:01.941592 3843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:01.962854 3843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:01.965284 3843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:01.965466 3843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:01.966003 3843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:01.966073 3843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:01.966141 3843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:01.966171 3843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:01.973155 3843 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:35727
I20260928 14:58:01.973224 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:35727 every 8 connection(s)
Sep 28 14:58:01 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:01.973616 3843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:01.977094 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:01.966594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:60869 (local address 127.1.252.254:41375)
0928 14:58:01.966678 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:01.966681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:01.966849 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:58:01.967357 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:01.967358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:01.967372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:01.967450 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:01.968420 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.969264 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.970900 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.971256 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.971347 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:01.973903 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:01.973905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:01.973905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:01.973916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:01.975589 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:01.976121 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:01.976122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:01.976123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:01.976185 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:01.976446 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:01.976447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:01.976448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:01.976567 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:01.976637 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:01.976922 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:58:01.977004 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:58:01.977293 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:01.966551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.1.252.193:60869)
0928 14:58:01.966712 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:01.966754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:01.967252 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:01.967573 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:01.967583 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:01.967990 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:01.968304 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:01.968310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.969371 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.969372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:01.970788 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:01.970792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.971365 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.971366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:01.971469 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:01.972169 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:01.972179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:01.973792 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:01.975702 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:01.975710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:01.975712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:01.976028 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:01.976253 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:01.976254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:01.976254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:01.976371 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:01.976641 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:01.976648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:01.976786 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:58:01.976893 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:58:01.977058 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260928 14:58:01.977836 3975 heartbeater.cc:347] Connected to a master server at 127.1.252.254:41375
I20260928 14:58:01.977958 3975 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:01.978139 3975 heartbeater.cc:511] Master 127.1.252.254:41375 requested a full tablet report, sending...
I20260928 14:58:01.978695 3770 ts_manager.cc:194] Registered new tserver with Master: 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727)
I20260928 14:58:01.979588 3770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:60869
I20260928 14:58:01.981977 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3843
I20260928 14:58:01.982084 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:01.982285 2035 external_mini_cluster.cc:1428] Setting key 63ce26ed0860a85f756c77e9c00fbc48
I20260928 14:58:01.984927 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:01.978862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.1.252.193:43549)
0928 14:58:01.979001 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:01.979011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:01.979070 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:01.979191 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:01.979191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:01.979332 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:01.979389 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:01.979390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.980203 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.980204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:01.980807 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:01.980809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.981169 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.981170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:01.981225 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:01.981595 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:01.981599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:01.982157 (+ 558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:01.983791 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:01.983792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:01.983793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:01.984063 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:01.984318 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:01.984319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:01.984320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:01.984393 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:01.984665 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:01.984665 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:01.984709 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:58:01.984759 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:01.984834 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 14:58:01.985013 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:01.978930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43549 (local address 127.1.252.254:41375)
0928 14:58:01.978988 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:01.978990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:01.979060 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:58:01.979123 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:01.979124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:01.979133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:01.979188 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:01.979452 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.980111 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.980899 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:01.981096 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:01.981160 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:01.982260 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:01.982261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:01.982262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:01.982268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:01.983706 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:01.984165 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:01.984166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:01.984167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:01.984236 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:01.984457 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:01.984458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:01.984458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:01.984600 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:01.984670 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:01.984806 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:58:01.984906 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:02.010381 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41375
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:02.089910 3984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:02.090101 3984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:02.090126 3984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:02.091177 3984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:02.091209 3984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:02.091229 3984 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:58:02.091241 3984 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:58:02.091259 3984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:02.092613 3984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:02.092792 3984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:02.092948 3984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:02.097909 3989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:02.097921 3990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:02.097919 3992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:02.098284 3984 server_base.cc:1034] running on GCE node
I20260928 14:58:02.098505 3984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:02.098731 3984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:02.099957 3984 hybrid_clock.cc:648] HybridClock initialized: now 1790607482099938 us; error 30 us; skew 500 ppm
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:02.102417 3984 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:02.102820 3984 webserver.cc:460] Webserver started at http://127.1.252.194:35089/ using document root <none> and password file <none>
I20260928 14:58:02.102996 3984 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:02.103036 3984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:02.103122 3984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:02.104166 3984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "4b07887caf7947589cb3152e1422bfc7"
format_stamp: "Formatted at 2026-09-28 14:58:02 on dist-test-slave-sf70"
server_key: "9209b946579781bc71fcdb07b262c47d"
server_key_iv: "fa7b6354da81eeae0586a42101d91c5f"
server_key_version: "encryptionkey@0"
I20260928 14:58:02.104465 3984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "4b07887caf7947589cb3152e1422bfc7"
format_stamp: "Formatted at 2026-09-28 14:58:02 on dist-test-slave-sf70"
server_key: "9209b946579781bc71fcdb07b262c47d"
server_key_iv: "fa7b6354da81eeae0586a42101d91c5f"
server_key_version: "encryptionkey@0"
I20260928 14:58:02.105803 3984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:02.106501 3999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.106693 3984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.106791 3984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "4b07887caf7947589cb3152e1422bfc7"
format_stamp: "Formatted at 2026-09-28 14:58:02 on dist-test-slave-sf70"
server_key: "9209b946579781bc71fcdb07b262c47d"
server_key_iv: "fa7b6354da81eeae0586a42101d91c5f"
server_key_version: "encryptionkey@0"
I20260928 14:58:02.106870 3984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:02.120118 3984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:02.122321 3984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:02.122494 3984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:02.122973 3984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:02.123030 3984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.123076 3984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:02.123106 3984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:02.130002 3984 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:38165
I20260928 14:58:02.130053 4115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:38165 every 8 connection(s)
I20260928 14:58:02.130330 3984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:02.132983 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.123500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:45395 (local address 127.1.252.254:41375)
0928 14:58:02.123578 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.123581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.123775 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:58:02.124319 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.124320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.124334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.124426 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:02.125239 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.126074 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.127680 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.128041 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.128120 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.130479 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:02.130480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:02.130481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:02.130488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:02.131808 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.132218 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.132219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.132219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.132271 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.132484 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.132485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.132485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.132583 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:02.132632 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.132850 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.132920 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:58:02.133164 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.123422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.1.252.194:45395)
0928 14:58:02.123666 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.123693 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.124218 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.124473 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.124476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.124835 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:02.125126 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.125131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.126191 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.126192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.127576 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.127579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.128136 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.128137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.128235 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.128782 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:02.128795 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:02.130334 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:02.131884 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.131929 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.131931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.132151 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.132336 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.132336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.132336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.132428 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.132638 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:02.132642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:02.132742 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.132827 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.132937 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260928 14:58:02.133600 4116 heartbeater.cc:347] Connected to a master server at 127.1.252.254:41375
I20260928 14:58:02.133677 4116 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:02.133883 4116 heartbeater.cc:511] Master 127.1.252.254:41375 requested a full tablet report, sending...
I20260928 14:58:02.134305 3771 ts_manager.cc:194] Registered new tserver with Master: 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165)
I20260928 14:58:02.134686 3771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:45395
I20260928 14:58:02.134990 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 3984
I20260928 14:58:02.135092 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:02.135272 2035 external_mini_cluster.cc:1428] Setting key b823936c7dbdab965bd6f12d9848ee57
I20260928 14:58:02.140672 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.134582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.1.252.194:40385)
0928 14:58:02.134702 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.134712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.134777 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.134934 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.134935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.135125 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:02.135178 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.135180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.136155 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.136156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.136656 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.136658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.137131 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.137132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.137209 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.137626 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:02.137632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:02.138014 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:02.139577 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.139578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.139579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.139804 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.140042 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.140043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.140043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.140095 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.140446 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:02.140447 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:02.140482 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.140532 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.140602 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260928 14:58:02.140767 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.134700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:40385 (local address 127.1.252.254:41375)
0928 14:58:02.134762 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.134765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.134775 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:02.134846 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.134847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.134858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.134923 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:02.135260 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.136066 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.136754 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.137010 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.137077 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.138103 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:02.138104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:02.138104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:02.138110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:02.139495 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.139907 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.139908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.139908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.139969 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.140160 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.140161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.140161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.140375 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:02.140454 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.140561 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.140642 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:02.163388 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41375
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:02.242622 4125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:02.242835 4125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:02.242859 4125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:02.243997 4125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:02.244040 4125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:02.244069 4125 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:58:02.244081 4125 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:58:02.244138 4125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:02.245651 4125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:02.245894 4125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:02.246105 4125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:02.251207 4133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:02.251286 4131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:02.251209 4130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:02.251585 4125 server_base.cc:1034] running on GCE node
I20260928 14:58:02.251799 4125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:02.252071 4125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:02.253233 4125 hybrid_clock.cc:648] HybridClock initialized: now 1790607482253225 us; error 41 us; skew 500 ppm
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:02.255668 4125 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:02.256132 4125 webserver.cc:460] Webserver started at http://127.1.252.195:46147/ using document root <none> and password file <none>
I20260928 14:58:02.256317 4125 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:02.256366 4125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:02.256451 4125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:02.257395 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "a26cac7b3a2d48499115c1c8189c839e"
format_stamp: "Formatted at 2026-09-28 14:58:02 on dist-test-slave-sf70"
server_key: "5d6d1e8e6994f0354dd8ea1a9af3822f"
server_key_iv: "82300e4a3a267cfa7d87d6c0b71af2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:02.257768 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "a26cac7b3a2d48499115c1c8189c839e"
format_stamp: "Formatted at 2026-09-28 14:58:02 on dist-test-slave-sf70"
server_key: "5d6d1e8e6994f0354dd8ea1a9af3822f"
server_key_iv: "82300e4a3a267cfa7d87d6c0b71af2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:02.259016 4125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:02.259946 4140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.260142 4125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:02.260237 4125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "a26cac7b3a2d48499115c1c8189c839e"
format_stamp: "Formatted at 2026-09-28 14:58:02 on dist-test-slave-sf70"
server_key: "5d6d1e8e6994f0354dd8ea1a9af3822f"
server_key_iv: "82300e4a3a267cfa7d87d6c0b71af2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:02.260308 4125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:02.285975 4125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:02.287973 4125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:02.288129 4125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:02.288630 4125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:02.288695 4125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.288739 4125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:02.288769 4125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.295614 4125 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35095
I20260928 14:58:02.295687 4256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35095 every 8 connection(s)
I20260928 14:58:02.296043 4125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:02.298230 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 4125
I20260928 14:58:02.298349 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:02.298533 2035 external_mini_cluster.cc:1428] Setting key 774734a443beda1f67f2c030b0d9a805
I20260928 14:58:02.299695 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.289223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:57953 (local address 127.1.252.254:41375)
0928 14:58:02.289266 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.289268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.289482 (+ 214us) server_negotiation.cc:407] Connection header received
0928 14:58:02.290048 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.290050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.290064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.290133 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:02.291094 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.291952 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.293662 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.293871 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.293924 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.296691 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:02.296692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:02.296693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:02.296699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:02.298128 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.298622 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.298623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.298624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.298714 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.299010 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.299011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.299011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.299158 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:02.299217 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.299567 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.299635 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:02.299921 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.289171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.1.252.195:57953)
0928 14:58:02.289342 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.289394 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.289935 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.290248 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.290254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.290673 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:02.290979 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.290987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.292088 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.292089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.293549 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.293553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.293952 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.293953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.294145 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.294675 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:02.294686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:02.296599 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:02.298231 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.298238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.298240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.298531 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.298778 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.298779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.298780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.298920 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.299228 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:02.299235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:02.299393 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.299525 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.299662 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260928 14:58:02.300365 4257 heartbeater.cc:347] Connected to a master server at 127.1.252.254:41375
I20260928 14:58:02.300448 4257 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:02.300623 4257 heartbeater.cc:511] Master 127.1.252.254:41375 requested a full tablet report, sending...
I20260928 14:58:02.301019 3771 ts_manager.cc:194] Registered new tserver with Master: a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:02.301513 3771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:57953
I20260928 14:58:02.307034 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.301494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:48441 (local address 127.1.252.254:41375)
0928 14:58:02.301544 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.301546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.301671 (+ 125us) server_negotiation.cc:407] Connection header received
0928 14:58:02.301736 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.301737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.301746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.301789 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:02.302101 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.302590 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.303331 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.303535 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.303617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.304530 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:02.304532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:02.304532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:02.304538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:02.305935 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.306283 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.306284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.306285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.306337 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.306540 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.306541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.306541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.306648 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:02.306721 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.306877 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.306946 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:58:02.307049 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.301400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.1.252.195:48441)
0928 14:58:02.301607 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.301619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.301679 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.301792 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.301792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.301964 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:02.302023 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.302024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.302682 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.302683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.303252 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.303254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.303620 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.303621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.303703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.304150 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:02.304155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:02.304423 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:02.306034 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.306035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.306035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.306211 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.306411 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.306411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.306412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.306469 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.306714 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:02.306715 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:02.306760 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.306846 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.306946 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":113,"threads_started":1}
I20260928 14:58:02.310124 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:02.332485 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.325884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.1.252.254:41375)
0928 14:58:02.325937 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.325939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.325982 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:58:02.326042 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.326043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.326052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.326101 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:02.326614 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.327111 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.327801 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.328038 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.328100 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.330196 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:02.330197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:02.330198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:02.330206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:02.331574 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.331859 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.331859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.331860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.331969 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.332120 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.332120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.332121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.332216 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:02.332285 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.332371 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.332425 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:58:02.334173 3771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:02.335938 3771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:02.342379 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.338213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35727 (local address 127.0.0.1:55174)
0928 14:58:02.338623 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.338634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.338720 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.339100 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.339101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.339103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.339147 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.339149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.340367 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.340368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.341233 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.341236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.342088 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.342089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.342163 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.342175 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.342245 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.342295 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":75,"threads_started":1}
I20260928 14:58:02.342458 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.338493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.1.252.193:35727)
0928 14:58:02.338641 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.338645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.338866 (+ 221us) server_negotiation.cc:407] Connection header received
0928 14:58:02.338940 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.338942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.338959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.339031 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.339224 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.340278 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.341313 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.341993 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.342072 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.342128 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.342273 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.342323 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260928 14:58:02.342600 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.338165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35095 (local address 127.0.0.1:53834)
0928 14:58:02.338360 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.338404 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.338473 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.338700 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.338703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.338705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.338775 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.338777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.340156 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.340157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.341458 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.341461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.342350 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.342351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.342441 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.342445 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.342480 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.342524 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":51,"thread_start_us":43,"threads_started":1}
I20260928 14:58:02.342911 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.338464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.1.252.194:38165)
0928 14:58:02.338825 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.338831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.338845 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:58:02.338875 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.338877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.338888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.338967 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.339245 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.340055 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.341562 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.342079 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.342144 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.342197 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.342417 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.342823 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":74,"threads_started":1}
I20260928 14:58:02.343006 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.338255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.1.252.195:35095)
0928 14:58:02.338517 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.338522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.338532 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:02.338569 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.338571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.338588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.338695 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.338851 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.340062 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.341544 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.342253 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.342768 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.342827 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.342851 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.342888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260928 14:58:02.343128 3909 tablet_service.cc:1467] Processing CreateTablet for tablet 14f7a277247b4d7389fa63faff5fec26 (DEFAULT_TABLE table=test-table [id=b04a9f935370477797794d55428d4a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:02.343487 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f7a277247b4d7389fa63faff5fec26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.343660 4191 tablet_service.cc:1467] Processing CreateTablet for tablet 14f7a277247b4d7389fa63faff5fec26 (DEFAULT_TABLE table=test-table [id=b04a9f935370477797794d55428d4a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:02.344045 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f7a277247b4d7389fa63faff5fec26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.342660 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.338347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38165 (local address 127.0.0.1:33872)
0928 14:58:02.338645 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.338656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.338710 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.339083 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.339084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.339087 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.339152 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.339154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.340135 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.340136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.341460 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.341461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.342241 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.342242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.342295 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.342298 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.342344 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.342374 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":44,"thread_start_us":68,"threads_started":1}
I20260928 14:58:02.346693 4280 tablet_bootstrap.cc:492] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Bootstrap starting.
I20260928 14:58:02.346779 4281 tablet_bootstrap.cc:492] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Bootstrap starting.
I20260928 14:58:02.347076 4050 tablet_service.cc:1467] Processing CreateTablet for tablet 14f7a277247b4d7389fa63faff5fec26 (DEFAULT_TABLE table=test-table [id=b04a9f935370477797794d55428d4a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:02.347340 4280 tablet_bootstrap.cc:654] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.347395 4050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f7a277247b4d7389fa63faff5fec26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.347636 4281 tablet_bootstrap.cc:654] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.347676 4280 log.cc:824] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:02.348019 4281 log.cc:824] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:02.348496 4280 tablet_bootstrap.cc:492] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: No bootstrap required, opened a new log
I20260928 14:58:02.348572 4280 ts_tablet_manager.cc:1397] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:02.348754 4281 tablet_bootstrap.cc:492] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: No bootstrap required, opened a new log
I20260928 14:58:02.348830 4281 ts_tablet_manager.cc:1397] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:02.350394 4281 raft_consensus.cc:348] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.350548 4281 raft_consensus.cc:374] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.350589 4281 raft_consensus.cc:729] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cac7b3a2d48499115c1c8189c839e, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.350661 4280 raft_consensus.cc:348] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.350731 4281 consensus_queue.cc:260] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.350826 4280 raft_consensus.cc:374] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.350871 4280 raft_consensus.cc:729] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.350976 4280 consensus_queue.cc:260] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.351068 4281 ts_tablet_manager.cc:1428] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:02.351191 4257 heartbeater.cc:503] Master 127.1.252.254:41375 was elected leader, sending a full tablet report...
I20260928 14:58:02.351250 4280 ts_tablet_manager.cc:1428] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:58:02.351370 3975 heartbeater.cc:503] Master 127.1.252.254:41375 was elected leader, sending a full tablet report...
I20260928 14:58:02.351480 4284 tablet_bootstrap.cc:492] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Bootstrap starting.
I20260928 14:58:02.352202 4284 tablet_bootstrap.cc:654] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.352540 4284 log.cc:824] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:02.353188 4284 tablet_bootstrap.cc:492] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: No bootstrap required, opened a new log
I20260928 14:58:02.353256 4284 ts_tablet_manager.cc:1397] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:02.354689 4284 raft_consensus.cc:348] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.354805 4284 raft_consensus.cc:374] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.354826 4284 raft_consensus.cc:729] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b07887caf7947589cb3152e1422bfc7, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.354933 4284 consensus_queue.cc:260] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.355134 4284 ts_tablet_manager.cc:1428] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:02.355222 4116 heartbeater.cc:503] Master 127.1.252.254:41375 was elected leader, sending a full tablet report...
W20260928 14:58:02.380952 4117 tablet.cc:2369] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:58:02.474524 3976 tablet.cc:2369] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:58:02.546736 4258 tablet.cc:2369] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:02.582680 4286 raft_consensus.cc:482] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:02.582844 4286 raft_consensus.cc:504] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.583216 4286 leader_election.cc:288] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095), 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165)
I20260928 14:58:02.586200 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.583445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35095 (local address 127.1.252.193:48265)
0928 14:58:02.583583 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.583595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.583649 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.583820 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.583821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.583824 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.583953 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.583956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.584599 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.584600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.585355 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.585357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.585925 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.585926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.586005 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.586010 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.586076 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.586119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260928 14:58:02.586189 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.583571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:48265 (local address 127.1.252.195:35095)
0928 14:58:02.583654 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.583656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.583673 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:58:02.583707 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.583708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.583718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.583812 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.584024 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.584513 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.585460 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.585831 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.585921 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.585930 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.586083 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.586120 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:58:02.586417 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.583769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:49347 (local address 127.1.252.194:38165)
0928 14:58:02.583820 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.583821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.583924 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:58:02.584007 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.584008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.584015 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.584090 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.584265 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.584737 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.585630 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.586008 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.586094 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.586102 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.586270 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.586324 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:58:02.586566 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f7a277247b4d7389fa63faff5fec26" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e" is_pre_election: true
I20260928 14:58:02.586632 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.583651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38165 (local address 127.1.252.193:49347)
0928 14:58:02.583833 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.583842 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.583949 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.584100 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.584100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.584102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.584180 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.584182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.584829 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.584830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.585545 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.585547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.586095 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.586096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.586168 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.586176 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.586504 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.586549 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 14:58:02.586706 4211 raft_consensus.cc:2454] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 0.
I20260928 14:58:02.586928 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f7a277247b4d7389fa63faff5fec26" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7" is_pre_election: true
I20260928 14:58:02.586957 3860 leader_election.cc:302] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c76786d2a924821a1b621a8c4d60605, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:02.587095 4070 raft_consensus.cc:2454] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 0.
I20260928 14:58:02.587129 4286 raft_consensus.cc:2781] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:02.587189 4286 raft_consensus.cc:482] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:02.587213 4286 raft_consensus.cc:3037] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.587989 4286 raft_consensus.cc:504] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.588099 4286 leader_election.cc:288] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 election: Requested vote from peers a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095), 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165)
I20260928 14:58:02.588304 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f7a277247b4d7389fa63faff5fec26" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7"
I20260928 14:58:02.588290 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f7a277247b4d7389fa63faff5fec26" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e"
I20260928 14:58:02.588382 4211 raft_consensus.cc:3037] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.588395 4070 raft_consensus.cc:3037] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.589047 4211 raft_consensus.cc:2454] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 1.
I20260928 14:58:02.589147 4070 raft_consensus.cc:2454] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 1.
I20260928 14:58:02.589223 3860 leader_election.cc:302] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c76786d2a924821a1b621a8c4d60605, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:02.589329 4286 raft_consensus.cc:2781] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:02.589479 4286 raft_consensus.cc:686] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 1 LEADER]: Becoming Leader. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Running, Role: LEADER
I20260928 14:58:02.589570 4286 consensus_queue.cc:237] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } }
I20260928 14:58:02.590348 3770 catalog_manager.cc:5495] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193). New cstate: current_term: 1 leader_uuid: "7c76786d2a924821a1b621a8c4d60605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } health_report { overall_health: HEALTHY } } }
Sep 28 14:58:02 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:02.659606 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.652671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.1.252.254:41375)
0928 14:58:02.652788 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.652791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.652907 (+ 116us) server_negotiation.cc:407] Connection header received
0928 14:58:02.652969 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.652970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.652980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.653060 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:02.653555 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.654064 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.654766 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.654954 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.655037 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.657023 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:02.657025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:02.657025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:02.657033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:02.658455 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.658830 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.658831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.658831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.658887 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:02.659085 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:02.659086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:02.659087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:02.659193 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:02.659266 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.659424 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.659525 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":83}
I20260928 14:58:02.659726 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.652608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41375 (local address 127.0.0.1:35566)
0928 14:58:02.652811 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.652823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.652910 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.653068 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.653068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.653413 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:02.653461 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.653465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.654145 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.654146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.654678 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.654680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.655039 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.655039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.655097 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.655628 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:02.655633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:02.656935 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:02.658554 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.658561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.658564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.658751 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.658955 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:02.658956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:02.658957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:02.659022 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:02.659255 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:02.659263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:02.659316 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.659402 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.659625 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":95,"thread_start_us":112,"threads_started":1}
I20260928 14:58:02.661659 3770 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:35566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:58:02.661895 3770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:02.663405 3909 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb74ad48fed4afc9820e7455491aba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01ea421255c345449b04568088d0deee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:02.663391 4050 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb74ad48fed4afc9820e7455491aba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01ea421255c345449b04568088d0deee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:02.663580 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb74ad48fed4afc9820e7455491aba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.663592 4050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb74ad48fed4afc9820e7455491aba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.664920 4284 tablet_bootstrap.cc:492] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: Bootstrap starting.
I20260928 14:58:02.664939 4280 tablet_bootstrap.cc:492] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: Bootstrap starting.
I20260928 14:58:02.665118 4191 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb74ad48fed4afc9820e7455491aba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01ea421255c345449b04568088d0deee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:02.665269 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb74ad48fed4afc9820e7455491aba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.665318 4280 tablet_bootstrap.cc:654] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.665333 4284 tablet_bootstrap.cc:654] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.665988 4284 tablet_bootstrap.cc:492] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: No bootstrap required, opened a new log
I20260928 14:58:02.666046 4284 ts_tablet_manager.cc:1397] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:02.666196 4284 raft_consensus.cc:348] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.666276 4284 raft_consensus.cc:374] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.666296 4284 raft_consensus.cc:729] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b07887caf7947589cb3152e1422bfc7, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.666352 4284 consensus_queue.cc:260] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.666460 4284 ts_tablet_manager.cc:1428] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:02.666602 4288 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.666687 4288 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.666972 4281 tablet_bootstrap.cc:492] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: Bootstrap starting.
I20260928 14:58:02.667073 4280 tablet_bootstrap.cc:492] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: No bootstrap required, opened a new log
I20260928 14:58:02.667127 4280 ts_tablet_manager.cc:1397] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:58:02.667488 4280 raft_consensus.cc:348] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.667543 4281 tablet_bootstrap.cc:654] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.667562 4280 raft_consensus.cc:374] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.667605 4280 raft_consensus.cc:729] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.667660 4280 consensus_queue.cc:260] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.667762 4280 ts_tablet_manager.cc:1428] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:02.667798 4286 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.667881 4286 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.668241 4281 tablet_bootstrap.cc:492] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: No bootstrap required, opened a new log
I20260928 14:58:02.668306 4281 ts_tablet_manager.cc:1397] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:02.668464 4281 raft_consensus.cc:348] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.668526 4281 raft_consensus.cc:374] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.668572 4281 raft_consensus.cc:729] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cac7b3a2d48499115c1c8189c839e, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.668622 4281 consensus_queue.cc:260] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.668705 4281 ts_tablet_manager.cc:1428] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.668776 4285 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.668870 4285 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.817142 4288 raft_consensus.cc:482] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:02.817243 4288 raft_consensus.cc:504] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.817552 4288 leader_election.cc:288] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727), a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:02.820746 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.817928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50117 (local address 127.1.252.195:35095)
0928 14:58:02.817973 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.817975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.818109 (+ 134us) server_negotiation.cc:407] Connection header received
0928 14:58:02.818187 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.818187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.818200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.818274 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.818437 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.818915 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.820003 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.820361 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.820446 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.820455 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.820617 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.820659 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:02.820762 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.817860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35095 (local address 127.1.252.194:50117)
0928 14:58:02.818016 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.818028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.818121 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.818280 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.818281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.818283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.818364 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.818366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.818997 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.818998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.819910 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.819913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.820471 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.820473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.820539 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.820542 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.820638 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.820672 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":47,"thread_start_us":73,"threads_started":1}
I20260928 14:58:02.821015 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb74ad48fed4afc9820e7455491aba5" candidate_uuid: "4b07887caf7947589cb3152e1422bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e" is_pre_election: true
I20260928 14:58:02.821127 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.817886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35727 (local address 127.1.252.194:50643)
0928 14:58:02.818040 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.818049 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.818132 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.818275 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.818276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.818278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:02.818368 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.818370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.819038 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.819039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.819856 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.819858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.820818 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.820820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.820904 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.820909 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.821004 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.821047 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":19,"thread_start_us":53,"threads_started":1}
I20260928 14:58:02.821120 4211 raft_consensus.cc:2454] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b07887caf7947589cb3152e1422bfc7 in term 0.
I20260928 14:58:02.821188 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.817971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50643 (local address 127.1.252.193:35727)
0928 14:58:02.818022 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.818024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.818116 (+ 92us) server_negotiation.cc:407] Connection header received
0928 14:58:02.818189 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.818190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.818199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.818273 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:02.818464 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.818946 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.819985 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.820342 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.820430 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.820439 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.821039 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.821089 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:58:02.821326 4001 leader_election.cc:302] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b07887caf7947589cb3152e1422bfc7, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:02.821460 4288 raft_consensus.cc:2781] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:02.821508 4288 raft_consensus.cc:482] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:02.821451 3929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb74ad48fed4afc9820e7455491aba5" candidate_uuid: "4b07887caf7947589cb3152e1422bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c76786d2a924821a1b621a8c4d60605" is_pre_election: true
I20260928 14:58:02.821529 4288 raft_consensus.cc:3037] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.821556 3929 raft_consensus.cc:2454] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b07887caf7947589cb3152e1422bfc7 in term 0.
I20260928 14:58:02.822319 4288 raft_consensus.cc:504] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.822490 4288 leader_election.cc:288] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [CANDIDATE]: Term 1 election: Requested vote from peers 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727), a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:02.822603 3929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb74ad48fed4afc9820e7455491aba5" candidate_uuid: "4b07887caf7947589cb3152e1422bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c76786d2a924821a1b621a8c4d60605"
I20260928 14:58:02.822649 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb74ad48fed4afc9820e7455491aba5" candidate_uuid: "4b07887caf7947589cb3152e1422bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e"
I20260928 14:58:02.822690 3929 raft_consensus.cc:3037] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.822703 4211 raft_consensus.cc:3037] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.823478 3929 raft_consensus.cc:2454] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b07887caf7947589cb3152e1422bfc7 in term 1.
I20260928 14:58:02.823478 4211 raft_consensus.cc:2454] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b07887caf7947589cb3152e1422bfc7 in term 1.
I20260928 14:58:02.823691 4001 leader_election.cc:302] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b07887caf7947589cb3152e1422bfc7, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:02.823803 4288 raft_consensus.cc:2781] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:02.823979 4288 raft_consensus.cc:686] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 4b07887caf7947589cb3152e1422bfc7, State: Running, Role: LEADER
I20260928 14:58:02.824118 4288 consensus_queue.cc:237] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.824570 4304 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: TxnStatusTablet state changed. Reason: New leader 4b07887caf7947589cb3152e1422bfc7. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.824662 4304 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:02.824838 4306 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:58:02.824950 4306 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:58:02.824992 3770 catalog_manager.cc:5495] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:02.850960 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.847985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.1.252.194:38165)
0928 14:58:02.848030 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.848033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.848089 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:58:02.848214 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.848215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.848235 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.848332 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:02.848463 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.849017 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.850107 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.850330 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.850394 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.850620 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.850726 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.850805 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.850870 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.850896 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:02.850996 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.847949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38165 (local address 127.0.0.1:33884)
0928 14:58:02.848036 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.848047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.848101 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.848331 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.848332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.848334 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:02.848387 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.848390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.849096 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.849097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.850009 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.850011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.850419 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.850420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.850489 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.850506 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.850801 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.850809 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.850867 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.850913 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:58:02.852691 4211 raft_consensus.cc:1261] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Refusing update from remote peer 4b07887caf7947589cb3152e1422bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:02.852717 3929 raft_consensus.cc:1261] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Refusing update from remote peer 4b07887caf7947589cb3152e1422bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:02.853008 4288 consensus_queue.cc:1035] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:02.853140 4304 consensus_queue.cc:1035] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:02.855125 4286 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: TxnStatusTablet state changed. Reason: New leader 4b07887caf7947589cb3152e1422bfc7. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.855217 4286 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.855580 4285 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: TxnStatusTablet state changed. Reason: New leader 4b07887caf7947589cb3152e1422bfc7. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.855667 4285 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.856316 4288 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.856401 4288 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:02.856477 4309 mvcc.cc:205] Tried to move back new op lower bound from 7334328249762197504 to 7334328249648177152. Current Snapshot: MvccSnapshot[applied={T|T < 7334328249762197504}]
I20260928 14:58:02.856655 4310 mvcc.cc:205] Tried to move back new op lower bound from 7334328249762197504 to 7334328249648177152. Current Snapshot: MvccSnapshot[applied={T|T < 7334328249762197504}]
I20260928 14:58:02.856724 4288 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.856786 4288 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 4b07887caf7947589cb3152e1422bfc7: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:02.857268 4286 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.857345 4286 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P 7c76786d2a924821a1b621a8c4d60605: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.857584 4312 mvcc.cc:205] Tried to move back new op lower bound from 7334328249762197504 to 7334328249648177152. Current Snapshot: MvccSnapshot[applied={T|T < 7334328249762197504}]
I20260928 14:58:02.857831 4285 tablet_replica.cc:439] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b07887caf7947589cb3152e1422bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } } }
I20260928 14:58:02.857923 4285 tablet_replica.cc:442] T 3eb74ad48fed4afc9820e7455491aba5 P a26cac7b3a2d48499115c1c8189c839e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:02.861804 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.858856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.1.252.193:35727)
0928 14:58:02.858900 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.858901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.859007 (+ 106us) server_negotiation.cc:407] Connection header received
0928 14:58:02.859086 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.859087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.859102 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.859172 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:02.859267 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.859785 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.861028 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.861267 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.861325 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.861497 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.861582 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.861645 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.861711 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.861728 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:02.870350 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.864223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43041 (local address 127.1.252.194:38165)
0928 14:58:02.864257 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.864259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.868091 (+ 3832us) server_negotiation.cc:407] Connection header received
0928 14:58:02.868164 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.868165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.868175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.868228 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:02.868378 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.868880 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.869640 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.869850 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.869922 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.870114 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.870152 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.870210 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.870261 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.870281 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 14:58:02.870414 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.864135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38165 (local address 127.1.252.193:43041)
0928 14:58:02.868019 (+ 3884us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.868030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.868115 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.868232 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.868233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.868234 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:02.868293 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.868295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.868962 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.868963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.869541 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.869543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.869922 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.869922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.870005 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.870017 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.870212 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.870215 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.870260 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.870309 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3841,"thread_start_us":71,"threads_started":1}
I20260928 14:58:02.872478 4211 raft_consensus.cc:1261] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Refusing update from remote peer 7c76786d2a924821a1b621a8c4d60605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:02.872534 4070 raft_consensus.cc:1261] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Refusing update from remote peer 7c76786d2a924821a1b621a8c4d60605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:02.872738 4291 consensus_queue.cc:1035] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:02.872836 4286 consensus_queue.cc:1035] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:02.881433 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.878889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:36733 (local address 127.1.252.193:35727)
0928 14:58:02.878926 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:02.878928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:02.879072 (+ 144us) server_negotiation.cc:407] Connection header received
0928 14:58:02.879139 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:02.879140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:02.879148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:02.879187 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:02.879318 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.879735 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.880847 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.881018 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.881068 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.881247 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.881285 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.881321 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:02.881375 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:58:02.881390 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:58:02.881505 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:02.878853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35727 (local address 127.1.252.194:36733)
0928 14:58:02.879021 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:02.879031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:02.879085 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:02.879191 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:02.879192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:02.879193 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:02.879265 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.879267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.879803 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.879804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.880757 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:02.880760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:02.881090 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:02.881091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:02.881159 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:02.881173 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:02.881324 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:02.881325 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:58:02.881374 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:02.881419 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260928 14:58:02.890604 3771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:02.890714 3771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:02.891469 3771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Sending DeleteTablet for 3 replicas of tablet 14f7a277247b4d7389fa63faff5fec26
I20260928 14:58:02.891762 4191 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f7a277247b4d7389fa63faff5fec26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:02 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 14:58:02.891841 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f7a277247b4d7389fa63faff5fec26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:02 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:55174
I20260928 14:58:02.892166 4050 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f7a277247b4d7389fa63faff5fec26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:02 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260928 14:58:02.892377 4332 tablet_replica.cc:330] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: stopping tablet replica
I20260928 14:58:02.892372 3771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:58:02.892481 4332 raft_consensus.cc:2229] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 1 LEADER]: Raft consensus shutting down.
W20260928 14:58:02.892517 3771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:02.892568 4332 raft_consensus.cc:2258] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:02.892961 4332 ts_tablet_manager.cc:1905] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:02.893913 4050 tablet_service.cc:1467] Processing CreateTablet for tablet e3beec6dda684a66b451dcc4376fe7a0 (DEFAULT_TABLE table=test-table [id=db7ebeb4dc924aebb50e314032dff1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:02.893949 4191 tablet_service.cc:1467] Processing CreateTablet for tablet e3beec6dda684a66b451dcc4376fe7a0 (DEFAULT_TABLE table=test-table [id=db7ebeb4dc924aebb50e314032dff1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:02.894032 4050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3beec6dda684a66b451dcc4376fe7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.894064 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3beec6dda684a66b451dcc4376fe7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.895443 4284 tablet_bootstrap.cc:492] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Bootstrap starting.
I20260928 14:58:02.895454 4281 tablet_bootstrap.cc:492] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Bootstrap starting.
I20260928 14:58:02.895737 4284 tablet_bootstrap.cc:654] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.895808 4281 tablet_bootstrap.cc:654] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.896138 3909 tablet_service.cc:1467] Processing CreateTablet for tablet e3beec6dda684a66b451dcc4376fe7a0 (DEFAULT_TABLE table=test-table [id=db7ebeb4dc924aebb50e314032dff1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:02.896240 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3beec6dda684a66b451dcc4376fe7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:02.897462 4280 tablet_bootstrap.cc:492] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Bootstrap starting.
I20260928 14:58:02.897619 4333 tablet_replica.cc:330] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: stopping tablet replica
I20260928 14:58:02.897710 4333 raft_consensus.cc:2229] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:02.897756 4333 raft_consensus.cc:2258] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:02.897890 4280 tablet_bootstrap.cc:654] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:02.898782 4332 ts_tablet_manager.cc:1918] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:02.898823 4333 ts_tablet_manager.cc:1905] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:02.898941 4332 log.cc:1196] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/14f7a277247b4d7389fa63faff5fec26
I20260928 14:58:02.899273 4332 ts_tablet_manager.cc:1939] T 14f7a277247b4d7389fa63faff5fec26 P 7c76786d2a924821a1b621a8c4d60605: Deleting consensus metadata
I20260928 14:58:02.899715 3755 catalog_manager.cc:4841] TS 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727): tablet 14f7a277247b4d7389fa63faff5fec26 (table test-table [id=b04a9f935370477797794d55428d4a5c]) successfully deleted
I20260928 14:58:02.900255 4280 tablet_bootstrap.cc:492] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: No bootstrap required, opened a new log
I20260928 14:58:02.900302 4280 ts_tablet_manager.cc:1397] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 14:58:02.900363 4331 tablet_replica.cc:330] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: stopping tablet replica
I20260928 14:58:02.900465 4331 raft_consensus.cc:2229] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:02.900462 4280 raft_consensus.cc:348] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.900516 4331 raft_consensus.cc:2258] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:02.900527 4280 raft_consensus.cc:374] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.900547 4280 raft_consensus.cc:729] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.900583 4280 consensus_queue.cc:260] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.900655 4280 ts_tablet_manager.cc:1428] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.901460 4333 ts_tablet_manager.cc:1918] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:02.901530 4333 log.cc:1196] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/14f7a277247b4d7389fa63faff5fec26
I20260928 14:58:02.901795 4333 ts_tablet_manager.cc:1939] T 14f7a277247b4d7389fa63faff5fec26 P 4b07887caf7947589cb3152e1422bfc7: Deleting consensus metadata
I20260928 14:58:02.900863 4331 ts_tablet_manager.cc:1905] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:02.903812 4281 tablet_bootstrap.cc:492] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: No bootstrap required, opened a new log
I20260928 14:58:02.903865 4281 ts_tablet_manager.cc:1397] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260928 14:58:02.903913 3756 catalog_manager.cc:4841] TS 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165): tablet 14f7a277247b4d7389fa63faff5fec26 (table test-table [id=b04a9f935370477797794d55428d4a5c]) successfully deleted
I20260928 14:58:02.904088 4284 tablet_bootstrap.cc:492] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: No bootstrap required, opened a new log
I20260928 14:58:02.904052 4281 raft_consensus.cc:348] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.904124 4284 ts_tablet_manager.cc:1397] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 14:58:02.904134 4281 raft_consensus.cc:374] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.904160 4281 raft_consensus.cc:729] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cac7b3a2d48499115c1c8189c839e, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.904210 4281 consensus_queue.cc:260] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.904255 4284 raft_consensus.cc:348] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.904294 4281 ts_tablet_manager.cc:1428] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.904327 4284 raft_consensus.cc:374] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:02.904350 4284 raft_consensus.cc:729] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b07887caf7947589cb3152e1422bfc7, State: Initialized, Role: FOLLOWER
I20260928 14:58:02.904399 4284 consensus_queue.cc:260] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.904488 4284 ts_tablet_manager.cc:1428] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:02.905015 4331 ts_tablet_manager.cc:1918] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:02.905086 4331 log.cc:1196] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/14f7a277247b4d7389fa63faff5fec26
I20260928 14:58:02.905342 4331 ts_tablet_manager.cc:1939] T 14f7a277247b4d7389fa63faff5fec26 P a26cac7b3a2d48499115c1c8189c839e: Deleting consensus metadata
I20260928 14:58:02.905736 3756 catalog_manager.cc:4841] TS a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095): tablet 14f7a277247b4d7389fa63faff5fec26 (table test-table [id=b04a9f935370477797794d55428d4a5c]) successfully deleted
I20260928 14:58:02.972494 4286 raft_consensus.cc:482] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:02.972606 4286 raft_consensus.cc:504] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.972750 4286 leader_election.cc:288] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165), a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:02.972976 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3beec6dda684a66b451dcc4376fe7a0" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7" is_pre_election: true
I20260928 14:58:02.973001 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3beec6dda684a66b451dcc4376fe7a0" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e" is_pre_election: true
I20260928 14:58:02.973088 4070 raft_consensus.cc:2454] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 0.
I20260928 14:58:02.973119 4211 raft_consensus.cc:2454] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 0.
I20260928 14:58:02.973306 3860 leader_election.cc:302] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b07887caf7947589cb3152e1422bfc7, 7c76786d2a924821a1b621a8c4d60605; no voters:
I20260928 14:58:02.973430 4286 raft_consensus.cc:2781] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:02.973487 4286 raft_consensus.cc:482] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:02.973510 4286 raft_consensus.cc:3037] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.974416 4286 raft_consensus.cc:504] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.974542 4286 leader_election.cc:288] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 election: Requested vote from peers 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165), a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:02.974727 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3beec6dda684a66b451dcc4376fe7a0" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7"
I20260928 14:58:02.974745 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3beec6dda684a66b451dcc4376fe7a0" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e"
I20260928 14:58:02.974795 4070 raft_consensus.cc:3037] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.974817 4211 raft_consensus.cc:3037] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:02.975517 4070 raft_consensus.cc:2454] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 1.
I20260928 14:58:02.975576 4211 raft_consensus.cc:2454] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 1.
I20260928 14:58:02.975729 3860 leader_election.cc:302] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c76786d2a924821a1b621a8c4d60605, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:02.975828 4286 raft_consensus.cc:2781] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:02.975930 4286 raft_consensus.cc:686] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 1 LEADER]: Becoming Leader. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Running, Role: LEADER
I20260928 14:58:02.975977 4286 consensus_queue.cc:237] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:02.976557 3771 catalog_manager.cc:5495] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193). New cstate: current_term: 1 leader_uuid: "7c76786d2a924821a1b621a8c4d60605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:02.995381 4070 raft_consensus.cc:1261] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Refusing update from remote peer 7c76786d2a924821a1b621a8c4d60605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:02.995381 4211 raft_consensus.cc:1261] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Refusing update from remote peer 7c76786d2a924821a1b621a8c4d60605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:02.995673 4291 consensus_queue.cc:1035] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:02.995790 4286 consensus_queue.cc:1035] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:58:03.000569 4312 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:58:04.519100 3770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:04.519193 3770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:04.520291 3770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Sending DeleteTablet for 3 replicas of tablet e3beec6dda684a66b451dcc4376fe7a0
I20260928 14:58:04.520675 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet e3beec6dda684a66b451dcc4376fe7a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:04 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:55174
I20260928 14:58:04.520737 4191 tablet_service.cc:1514] Processing DeleteTablet for tablet e3beec6dda684a66b451dcc4376fe7a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:04 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 14:58:04.520910 4050 tablet_service.cc:1514] Processing DeleteTablet for tablet e3beec6dda684a66b451dcc4376fe7a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:04 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260928 14:58:04.521253 4362 tablet_replica.cc:330] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: stopping tablet replica
I20260928 14:58:04.521310 4362 raft_consensus.cc:2229] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:04.521349 4362 raft_consensus.cc:2258] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:04.521526 4360 tablet_replica.cc:330] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: stopping tablet replica
I20260928 14:58:04.521575 4360 raft_consensus.cc:2229] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:04.521632 4360 raft_consensus.cc:2258] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:04.521884 4360 ts_tablet_manager.cc:1905] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:04.522742 4361 tablet_replica.cc:330] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: stopping tablet replica
I20260928 14:58:04.522821 4361 raft_consensus.cc:2229] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:04.522873 4361 raft_consensus.cc:2258] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:04.523212 4361 ts_tablet_manager.cc:1905] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:04.523873 4360 ts_tablet_manager.cc:1918] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:58:04.523988 4360 log.cc:1196] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/e3beec6dda684a66b451dcc4376fe7a0
I20260928 14:58:04.524299 4360 ts_tablet_manager.cc:1939] T e3beec6dda684a66b451dcc4376fe7a0 P 7c76786d2a924821a1b621a8c4d60605: Deleting consensus metadata
I20260928 14:58:04.524737 4362 ts_tablet_manager.cc:1905] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:04.524734 3755 catalog_manager.cc:4841] TS 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727): tablet e3beec6dda684a66b451dcc4376fe7a0 (table test-table [id=db7ebeb4dc924aebb50e314032dff1bf]) successfully deleted
I20260928 14:58:04.525341 4361 ts_tablet_manager.cc:1918] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:58:04.525408 4361 log.cc:1196] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/e3beec6dda684a66b451dcc4376fe7a0
I20260928 14:58:04.525656 4361 ts_tablet_manager.cc:1939] T e3beec6dda684a66b451dcc4376fe7a0 P a26cac7b3a2d48499115c1c8189c839e: Deleting consensus metadata
I20260928 14:58:04.526170 3756 catalog_manager.cc:4841] TS a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095): tablet e3beec6dda684a66b451dcc4376fe7a0 (table test-table [id=db7ebeb4dc924aebb50e314032dff1bf]) successfully deleted
I20260928 14:58:04.526839 4362 ts_tablet_manager.cc:1918] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:58:04.526906 4362 log.cc:1196] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/e3beec6dda684a66b451dcc4376fe7a0
I20260928 14:58:04.527181 4362 ts_tablet_manager.cc:1939] T e3beec6dda684a66b451dcc4376fe7a0 P 4b07887caf7947589cb3152e1422bfc7: Deleting consensus metadata
I20260928 14:58:04.527572 3756 catalog_manager.cc:4841] TS 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165): tablet e3beec6dda684a66b451dcc4376fe7a0 (table test-table [id=db7ebeb4dc924aebb50e314032dff1bf]) successfully deleted
Sep 28 14:58:04 dist-test-slave-sf70 krb5kdc[3725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:58:04 dist-test-slave-sf70 krb5kdc[3725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:04.547264 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.539519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.1.252.254:41375)
0928 14:58:04.539658 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.539660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.539747 (+ 87us) server_negotiation.cc:407] Connection header received
0928 14:58:04.539833 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.539834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.539848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.540117 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:04.540697 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.541290 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.542080 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.542305 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.542391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.544825 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:04.544826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:04.544827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:04.544833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:04.546361 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:04.546664 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:04.546665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:04.546665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:04.546719 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:04.546866 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:04.546866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:04.546866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:04.546965 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:04.547037 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.547121 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.547181 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260928 14:58:04.548951 3770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:04.549098 3770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:04.550699 3909 tablet_service.cc:1467] Processing CreateTablet for tablet 641bccbbf0e647f7b26759d3bd907236 (DEFAULT_TABLE table=test-table [id=78f466be7a104088ac852c8b3a728452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:04.550726 4050 tablet_service.cc:1467] Processing CreateTablet for tablet 641bccbbf0e647f7b26759d3bd907236 (DEFAULT_TABLE table=test-table [id=78f466be7a104088ac852c8b3a728452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:04.550860 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641bccbbf0e647f7b26759d3bd907236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:04.550884 4050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641bccbbf0e647f7b26759d3bd907236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:04.552069 4371 tablet_bootstrap.cc:492] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Bootstrap starting.
I20260928 14:58:04.552443 4371 tablet_bootstrap.cc:654] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:04.552532 4372 tablet_bootstrap.cc:492] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Bootstrap starting.
I20260928 14:58:04.552592 4191 tablet_service.cc:1467] Processing CreateTablet for tablet 641bccbbf0e647f7b26759d3bd907236 (DEFAULT_TABLE table=test-table [id=78f466be7a104088ac852c8b3a728452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:04.552716 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641bccbbf0e647f7b26759d3bd907236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:04.552915 4372 tablet_bootstrap.cc:654] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:04.553067 4371 tablet_bootstrap.cc:492] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: No bootstrap required, opened a new log
I20260928 14:58:04.553128 4371 ts_tablet_manager.cc:1397] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:04.553273 4371 raft_consensus.cc:348] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.553351 4371 raft_consensus.cc:374] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:04.553378 4371 raft_consensus.cc:729] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Initialized, Role: FOLLOWER
I20260928 14:58:04.553423 4371 consensus_queue.cc:260] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.553521 4371 ts_tablet_manager.cc:1428] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:04.554036 4375 tablet_bootstrap.cc:492] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Bootstrap starting.
I20260928 14:58:04.554576 4372 tablet_bootstrap.cc:492] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: No bootstrap required, opened a new log
I20260928 14:58:04.554574 4375 tablet_bootstrap.cc:654] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:04.554637 4372 ts_tablet_manager.cc:1397] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:04.554754 4372 raft_consensus.cc:348] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.554839 4372 raft_consensus.cc:374] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:04.554864 4372 raft_consensus.cc:729] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b07887caf7947589cb3152e1422bfc7, State: Initialized, Role: FOLLOWER
I20260928 14:58:04.554914 4372 consensus_queue.cc:260] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.555037 4372 ts_tablet_manager.cc:1428] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:04.555269 4375 tablet_bootstrap.cc:492] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: No bootstrap required, opened a new log
I20260928 14:58:04.555332 4375 ts_tablet_manager.cc:1397] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:04.555455 4375 raft_consensus.cc:348] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.555532 4375 raft_consensus.cc:374] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:04.555553 4375 raft_consensus.cc:729] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cac7b3a2d48499115c1c8189c839e, State: Initialized, Role: FOLLOWER
I20260928 14:58:04.555594 4375 consensus_queue.cc:260] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.555796 4375 ts_tablet_manager.cc:1428] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:04.692651 4354 raft_consensus.cc:482] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:04.692760 4354 raft_consensus.cc:504] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.692945 4354 leader_election.cc:288] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165), a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:04.693235 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641bccbbf0e647f7b26759d3bd907236" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7" is_pre_election: true
I20260928 14:58:04.693256 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641bccbbf0e647f7b26759d3bd907236" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e" is_pre_election: true
I20260928 14:58:04.693354 4211 raft_consensus.cc:2454] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 0.
I20260928 14:58:04.693354 4070 raft_consensus.cc:2454] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 0.
I20260928 14:58:04.693526 3860 leader_election.cc:302] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b07887caf7947589cb3152e1422bfc7, 7c76786d2a924821a1b621a8c4d60605; no voters:
I20260928 14:58:04.693634 4354 raft_consensus.cc:2781] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:04.693673 4354 raft_consensus.cc:482] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:04.693693 4354 raft_consensus.cc:3037] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:04.694406 4354 raft_consensus.cc:504] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.694545 4354 leader_election.cc:288] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 election: Requested vote from peers 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165), a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095)
I20260928 14:58:04.694710 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641bccbbf0e647f7b26759d3bd907236" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7"
I20260928 14:58:04.694726 4211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641bccbbf0e647f7b26759d3bd907236" candidate_uuid: "7c76786d2a924821a1b621a8c4d60605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26cac7b3a2d48499115c1c8189c839e"
I20260928 14:58:04.694787 4070 raft_consensus.cc:3037] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:04.694789 4211 raft_consensus.cc:3037] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:04.695508 4211 raft_consensus.cc:2454] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 1.
I20260928 14:58:04.695508 4070 raft_consensus.cc:2454] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76786d2a924821a1b621a8c4d60605 in term 1.
I20260928 14:58:04.695744 3860 leader_election.cc:302] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b07887caf7947589cb3152e1422bfc7, 7c76786d2a924821a1b621a8c4d60605; no voters:
I20260928 14:58:04.695865 4354 raft_consensus.cc:2781] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:04.695962 4354 raft_consensus.cc:686] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 1 LEADER]: Becoming Leader. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Running, Role: LEADER
I20260928 14:58:04.696020 4354 consensus_queue.cc:237] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.696803 3770 catalog_manager.cc:5495] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193). New cstate: current_term: 1 leader_uuid: "7c76786d2a924821a1b621a8c4d60605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:04.738312 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.736011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.1.252.193:35727)
0928 14:58:04.736170 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.736171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.736179 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:04.736199 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.736199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.736209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.736276 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:04.736395 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.736894 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.737619 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.737847 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.737901 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.738034 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:04.738077 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:04.738140 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.738200 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.738222 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260928 14:58:04.740659 4211 raft_consensus.cc:1261] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Refusing update from remote peer 7c76786d2a924821a1b621a8c4d60605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:04.740686 4070 raft_consensus.cc:1261] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Refusing update from remote peer 7c76786d2a924821a1b621a8c4d60605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:04.740893 4351 consensus_queue.cc:1035] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:04.741029 4354 consensus_queue.cc:1035] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:04.751060 3771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:04.751173 3771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:04.751804 3771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Sending DeleteTablet for 3 replicas of tablet 641bccbbf0e647f7b26759d3bd907236
I20260928 14:58:04.752040 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 641bccbbf0e647f7b26759d3bd907236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:04 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:55174
I20260928 14:58:04.752096 4050 tablet_service.cc:1514] Processing DeleteTablet for tablet 641bccbbf0e647f7b26759d3bd907236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:04 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260928 14:58:04.752137 4360 tablet_replica.cc:330] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: stopping tablet replica
I20260928 14:58:04.752174 4362 tablet_replica.cc:330] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: stopping tablet replica
I20260928 14:58:04.752187 4360 raft_consensus.cc:2229] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:04.752161 4191 tablet_service.cc:1514] Processing DeleteTablet for tablet 641bccbbf0e647f7b26759d3bd907236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:04 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 14:58:04.752220 4362 raft_consensus.cc:2229] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:04.752238 4360 raft_consensus.cc:2258] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:04.752256 4362 raft_consensus.cc:2258] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:04.752589 4360 ts_tablet_manager.cc:1905] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:04.752667 4362 ts_tablet_manager.cc:1905] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:04.752751 4361 tablet_replica.cc:330] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: stopping tablet replica
I20260928 14:58:04.752820 4361 raft_consensus.cc:2229] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:04.752871 4361 raft_consensus.cc:2258] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:04.753149 4361 ts_tablet_manager.cc:1905] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:04.754122 4360 ts_tablet_manager.cc:1918] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:04.754187 4360 log.cc:1196] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/641bccbbf0e647f7b26759d3bd907236
I20260928 14:58:04.754347 3771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:58:04.754439 4362 ts_tablet_manager.cc:1918] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:04.754462 4360 ts_tablet_manager.cc:1939] T 641bccbbf0e647f7b26759d3bd907236 P 7c76786d2a924821a1b621a8c4d60605: Deleting consensus metadata
I20260928 14:58:04.754490 4362 log.cc:1196] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/641bccbbf0e647f7b26759d3bd907236
W20260928 14:58:04.754499 3771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:04.754698 4362 ts_tablet_manager.cc:1939] T 641bccbbf0e647f7b26759d3bd907236 P 4b07887caf7947589cb3152e1422bfc7: Deleting consensus metadata
I20260928 14:58:04.754735 4361 ts_tablet_manager.cc:1918] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:04.754765 3755 catalog_manager.cc:4841] TS 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727): tablet 641bccbbf0e647f7b26759d3bd907236 (table test-table [id=78f466be7a104088ac852c8b3a728452]) successfully deleted
I20260928 14:58:04.754783 4361 log.cc:1196] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/641bccbbf0e647f7b26759d3bd907236
I20260928 14:58:04.755012 4361 ts_tablet_manager.cc:1939] T 641bccbbf0e647f7b26759d3bd907236 P a26cac7b3a2d48499115c1c8189c839e: Deleting consensus metadata
I20260928 14:58:04.755527 3756 catalog_manager.cc:4841] TS 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165): tablet 641bccbbf0e647f7b26759d3bd907236 (table test-table [id=78f466be7a104088ac852c8b3a728452]) successfully deleted
I20260928 14:58:04.755590 3756 catalog_manager.cc:4841] TS a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095): tablet 641bccbbf0e647f7b26759d3bd907236 (table test-table [id=78f466be7a104088ac852c8b3a728452]) successfully deleted
I20260928 14:58:04.755870 3909 tablet_service.cc:1467] Processing CreateTablet for tablet d9f800dff00a4065b78ef920dc60e57b (DEFAULT_TABLE table=test-table [id=cb4e4a8ca47a4f9a8bb56996b99f7a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:04.755934 4050 tablet_service.cc:1467] Processing CreateTablet for tablet d9f800dff00a4065b78ef920dc60e57b (DEFAULT_TABLE table=test-table [id=cb4e4a8ca47a4f9a8bb56996b99f7a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:04.756004 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f800dff00a4065b78ef920dc60e57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:04.755983 4191 tablet_service.cc:1467] Processing CreateTablet for tablet d9f800dff00a4065b78ef920dc60e57b (DEFAULT_TABLE table=test-table [id=cb4e4a8ca47a4f9a8bb56996b99f7a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:04.756016 4050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f800dff00a4065b78ef920dc60e57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:04.756089 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f800dff00a4065b78ef920dc60e57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:04.756865 4375 tablet_bootstrap.cc:492] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Bootstrap starting.
I20260928 14:58:04.757140 4372 tablet_bootstrap.cc:492] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Bootstrap starting.
I20260928 14:58:04.757153 4371 tablet_bootstrap.cc:492] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605: Bootstrap starting.
I20260928 14:58:04.757472 4375 tablet_bootstrap.cc:654] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:04.757488 4371 tablet_bootstrap.cc:654] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:04.757491 4372 tablet_bootstrap.cc:654] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:04.758106 4375 tablet_bootstrap.cc:492] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: No bootstrap required, opened a new log
I20260928 14:58:04.758157 4375 ts_tablet_manager.cc:1397] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:04.758164 4372 tablet_bootstrap.cc:492] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: No bootstrap required, opened a new log
I20260928 14:58:04.758209 4372 ts_tablet_manager.cc:1397] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:04.758266 4371 tablet_bootstrap.cc:492] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605: No bootstrap required, opened a new log
I20260928 14:58:04.758318 4371 ts_tablet_manager.cc:1397] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:04.758306 4375 raft_consensus.cc:348] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.758396 4375 raft_consensus.cc:374] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:04.758379 4372 raft_consensus.cc:348] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.758422 4375 raft_consensus.cc:729] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cac7b3a2d48499115c1c8189c839e, State: Initialized, Role: FOLLOWER
I20260928 14:58:04.758457 4372 raft_consensus.cc:374] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:04.758421 4371 raft_consensus.cc:348] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.758484 4372 raft_consensus.cc:729] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b07887caf7947589cb3152e1422bfc7, State: Initialized, Role: FOLLOWER
I20260928 14:58:04.758486 4371 raft_consensus.cc:374] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:04.758508 4371 raft_consensus.cc:729] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76786d2a924821a1b621a8c4d60605, State: Initialized, Role: FOLLOWER
I20260928 14:58:04.758478 4375 consensus_queue.cc:260] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.758529 4372 consensus_queue.cc:260] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.758574 4375 ts_tablet_manager.cc:1428] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:04.758553 4371 consensus_queue.cc:260] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.758621 4372 ts_tablet_manager.cc:1428] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:04.758651 4371 ts_tablet_manager.cc:1428] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:04.851177 4377 raft_consensus.cc:482] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:04.851267 4377 raft_consensus.cc:504] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.851763 4377 leader_election.cc:288] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727), 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165)
I20260928 14:58:04.854789 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.852038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:49963 (local address 127.1.252.193:35727)
0928 14:58:04.852086 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.852089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.852195 (+ 106us) server_negotiation.cc:407] Connection header received
0928 14:58:04.852272 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.852273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.852282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.852369 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:04.852549 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.853056 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.854042 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.854419 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.854504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.854514 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.854665 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.854705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:58:04.854803 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.851948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35727 (local address 127.1.252.195:49963)
0928 14:58:04.852109 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:04.852120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:04.852209 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:04.852375 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:04.852376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:04.852378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:04.852476 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.852478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.853142 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.853143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.853945 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.853947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.854503 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.854504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.854581 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.854584 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:04.854646 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:58:04.854686 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260928 14:58:04.854820 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.852071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:37061 (local address 127.1.252.194:38165)
0928 14:58:04.852183 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.852185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.852261 (+ 76us) server_negotiation.cc:407] Connection header received
0928 14:58:04.852323 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.852324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.852335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.852426 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:04.852605 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.853084 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.854053 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.854444 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.854530 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.854539 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.854691 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.854734 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260928 14:58:04.854827 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.852031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38165 (local address 127.1.252.195:37061)
0928 14:58:04.852180 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:04.852191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:04.852259 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:04.852430 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:04.852431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:04.852433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:04.852530 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.852532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.853187 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.853188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.853955 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.853957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.854528 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.854529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.854613 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.854617 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:04.854685 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:58:04.854727 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260928 14:58:04.855070 3929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f800dff00a4065b78ef920dc60e57b" candidate_uuid: "a26cac7b3a2d48499115c1c8189c839e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c76786d2a924821a1b621a8c4d60605" is_pre_election: true
I20260928 14:58:04.855175 3929 raft_consensus.cc:2454] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a26cac7b3a2d48499115c1c8189c839e in term 0.
I20260928 14:58:04.855235 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f800dff00a4065b78ef920dc60e57b" candidate_uuid: "a26cac7b3a2d48499115c1c8189c839e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7" is_pre_election: true
I20260928 14:58:04.855332 4070 raft_consensus.cc:2454] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a26cac7b3a2d48499115c1c8189c839e in term 0.
I20260928 14:58:04.855350 4141 leader_election.cc:302] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c76786d2a924821a1b621a8c4d60605, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:04.855477 4377 raft_consensus.cc:2781] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:04.855528 4377 raft_consensus.cc:482] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:04.855559 4377 raft_consensus.cc:3037] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:04.856283 4377 raft_consensus.cc:504] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.856464 4377 leader_election.cc:288] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [CANDIDATE]: Term 1 election: Requested vote from peers 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727), 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165)
I20260928 14:58:04.856602 3929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f800dff00a4065b78ef920dc60e57b" candidate_uuid: "a26cac7b3a2d48499115c1c8189c839e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c76786d2a924821a1b621a8c4d60605"
I20260928 14:58:04.856628 4070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f800dff00a4065b78ef920dc60e57b" candidate_uuid: "a26cac7b3a2d48499115c1c8189c839e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b07887caf7947589cb3152e1422bfc7"
I20260928 14:58:04.856698 3929 raft_consensus.cc:3037] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:04.856701 4070 raft_consensus.cc:3037] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:04.857299 3929 raft_consensus.cc:2454] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a26cac7b3a2d48499115c1c8189c839e in term 1.
I20260928 14:58:04.857311 4070 raft_consensus.cc:2454] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a26cac7b3a2d48499115c1c8189c839e in term 1.
I20260928 14:58:04.857484 4142 leader_election.cc:302] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b07887caf7947589cb3152e1422bfc7, a26cac7b3a2d48499115c1c8189c839e; no voters:
I20260928 14:58:04.857610 4377 raft_consensus.cc:2781] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:04.857745 4377 raft_consensus.cc:686] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 1 LEADER]: Becoming Leader. State: Replica: a26cac7b3a2d48499115c1c8189c839e, State: Running, Role: LEADER
I20260928 14:58:04.857828 4377 consensus_queue.cc:237] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } }
I20260928 14:58:04.858544 3770 catalog_manager.cc:5495] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e reported cstate change: term changed from 0 to 1, leader changed from <none> to a26cac7b3a2d48499115c1c8189c839e (127.1.252.195). New cstate: current_term: 1 leader_uuid: "a26cac7b3a2d48499115c1c8189c839e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a26cac7b3a2d48499115c1c8189c839e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35095 } health_report { overall_health: HEALTHY } } }
I20260928 14:58:04.879113 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.876458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.1.252.195:35095)
0928 14:58:04.876650 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.876651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.876658 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:58:04.876691 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.876691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.876709 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.876802 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:04.876911 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.877477 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.878252 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.878513 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.878590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.878729 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:04.878844 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:04.878949 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.878990 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.879009 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":106,"threads_started":1}
I20260928 14:58:04.884253 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.881762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:54259 (local address 127.1.252.194:38165)
0928 14:58:04.881862 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.881864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.882005 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:58:04.882030 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.882031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.882040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.882087 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:04.882205 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.882652 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.883495 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.883670 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.883722 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.884017 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:04.884048 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:04.884095 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.884159 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.884197 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:58:04.884284 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.881640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38165 (local address 127.1.252.195:54259)
0928 14:58:04.881933 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:04.881942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:04.881996 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:04.882104 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:04.882104 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:04.882106 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:04.882152 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.882153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.882744 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.882744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.883416 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.883419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.883806 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.883807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.883861 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.883874 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:04.884111 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:04.884113 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:04.884155 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 14:58:04.884192 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":142,"threads_started":1}
I20260928 14:58:04.887012 3929 raft_consensus.cc:1261] T d9f800dff00a4065b78ef920dc60e57b P 7c76786d2a924821a1b621a8c4d60605 [term 1 FOLLOWER]: Refusing update from remote peer a26cac7b3a2d48499115c1c8189c839e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:04.887063 4070 raft_consensus.cc:1261] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Refusing update from remote peer a26cac7b3a2d48499115c1c8189c839e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:04.887331 4377 consensus_queue.cc:1035] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b07887caf7947589cb3152e1422bfc7" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:04.887490 4391 consensus_queue.cc:1035] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c76786d2a924821a1b621a8c4d60605" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:04.895707 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.892927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:59217 (local address 127.1.252.195:35095)
0928 14:58:04.892981 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:04.892983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:04.893101 (+ 118us) server_negotiation.cc:407] Connection header received
0928 14:58:04.893171 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:04.893172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:04.893183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:04.893256 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:04.893379 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.893897 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.894906 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.895183 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.895287 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.895530 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:04.895558 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:04.895594 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:04.895637 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:58:04.895655 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:58:04.895732 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:04.892880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35095 (local address 127.1.252.194:59217)
0928 14:58:04.893005 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:04.893051 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:04.893108 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:04.893248 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:04.893249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:04.893250 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:04.893321 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.893323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.893983 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.893984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.894651 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:04.894654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:04.895297 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:04.895298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:04.895444 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:04.895452 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:04.895602 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:04.895604 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:04.895634 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 14:58:04.895674 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":84,"threads_started":1}
W20260928 14:58:04.896224 4343 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:58:06.422571 3770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:06.422672 3770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:06.423450 3770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d085be63bff847e4bbe45145e90c7918: Sending DeleteTablet for 3 replicas of tablet d9f800dff00a4065b78ef920dc60e57b
I20260928 14:58:06.423683 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f800dff00a4065b78ef920dc60e57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:06 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:55174
I20260928 14:58:06.423694 4049 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f800dff00a4065b78ef920dc60e57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:06 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260928 14:58:06.423744 4191 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f800dff00a4065b78ef920dc60e57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:06 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 14:58:06.423949 4428 tablet_replica.cc:330] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: stopping tablet replica
I20260928 14:58:06.424001 4427 tablet_replica.cc:330] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: stopping tablet replica
I20260928 14:58:06.424026 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3843
I20260928 14:58:06.424053 4427 raft_consensus.cc:2229] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:06.424571 4427 raft_consensus.cc:2258] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:06.424825 4427 ts_tablet_manager.cc:1905] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:06.425225 4428 raft_consensus.cc:2229] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:06.425324 4428 raft_consensus.cc:2258] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:06.425552 4428 ts_tablet_manager.cc:1905] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:06.426734 4427 ts_tablet_manager.cc:1918] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:58:06.426807 4427 log.cc:1196] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/d9f800dff00a4065b78ef920dc60e57b
I20260928 14:58:06.427045 4427 ts_tablet_manager.cc:1939] T d9f800dff00a4065b78ef920dc60e57b P 4b07887caf7947589cb3152e1422bfc7: Deleting consensus metadata
I20260928 14:58:06.427394 4428 ts_tablet_manager.cc:1918] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:58:06.427452 4428 log.cc:1196] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/d9f800dff00a4065b78ef920dc60e57b
I20260928 14:58:06.427503 3756 catalog_manager.cc:4841] TS 4b07887caf7947589cb3152e1422bfc7 (127.1.252.194:38165): tablet d9f800dff00a4065b78ef920dc60e57b (table test-table [id=cb4e4a8ca47a4f9a8bb56996b99f7a1e]) successfully deleted
I20260928 14:58:06.427659 4428 ts_tablet_manager.cc:1939] T d9f800dff00a4065b78ef920dc60e57b P a26cac7b3a2d48499115c1c8189c839e: Deleting consensus metadata
I20260928 14:58:06.427981 3756 catalog_manager.cc:4841] TS a26cac7b3a2d48499115c1c8189c839e (127.1.252.195:35095): tablet d9f800dff00a4065b78ef920dc60e57b (table test-table [id=cb4e4a8ca47a4f9a8bb56996b99f7a1e]) successfully deleted
W20260928 14:58:06.431187 3755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:35727 (error 108)
I20260928 14:58:06.431445 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3984
I20260928 14:58:06.432176 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:06.431582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35727 (local address 127.0.0.1:55196)
0928 14:58:06.432105 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:06.432146 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":55,"threads_started":1}
W20260928 14:58:06.432291 3755 catalog_manager.cc:4584] TS 7c76786d2a924821a1b621a8c4d60605 (127.1.252.193:35727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9f800dff00a4065b78ef920dc60e57b: Network error: Client connection negotiation failed: client connection to 127.1.252.193:35727: connect: Connection refused (error 111)
I20260928 14:58:06.437790 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 4125
I20260928 14:58:06.443367 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 3738
2026-09-28T14:58:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5961 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:06 dist-test-slave-sf70 krb5kdc[4433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:06 dist-test-slave-sf70 krb5kdc[4433](info): set up 2 sockets
Sep 28 14:58:06 dist-test-slave-sf70 krb5kdc[4433](info): commencing operation
krb5kdc: starting...
W20260928 14:58:07.485741 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:07 dist-test-slave-sf70 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:07.606700 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37403
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:07.685652 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:07.685889 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:07.685927 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:07.687047 4446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:07.687116 4446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:07.687141 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:07.687161 4446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:07.687193 4446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:07.688634 4446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:07.688905 4446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:07.689116 4446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:07.694054 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:07.694213 4446 server_base.cc:1034] running on GCE node
W20260928 14:58:07.694187 4454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:07.694187 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:07.694625 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:07.694907 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:07.696076 4446 hybrid_clock.cc:648] HybridClock initialized: now 1790607487696062 us; error 29 us; skew 500 ppm
Sep 28 14:58:07 dist-test-slave-sf70 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:07.698561 4446 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:07.698999 4446 webserver.cc:460] Webserver started at http://127.1.252.254:43247/ using document root <none> and password file <none>
I20260928 14:58:07.699229 4446 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:07.699295 4446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:07.699429 4446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:07.700523 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "40e86bdcf5584312a7761b3ba4f162ea"
format_stamp: "Formatted at 2026-09-28 14:58:07 on dist-test-slave-sf70"
server_key: "f4a36bc45c6030ea9efc3255ade9c93b"
server_key_iv: "85ba25c6b69da86501611d930411ec01"
server_key_version: "encryptionkey@0"
I20260928 14:58:07.700889 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "40e86bdcf5584312a7761b3ba4f162ea"
format_stamp: "Formatted at 2026-09-28 14:58:07 on dist-test-slave-sf70"
server_key: "f4a36bc45c6030ea9efc3255ade9c93b"
server_key_iv: "85ba25c6b69da86501611d930411ec01"
server_key_version: "encryptionkey@0"
I20260928 14:58:07.702224 4446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:07.702955 4461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:07.703122 4446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:07.703234 4446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "40e86bdcf5584312a7761b3ba4f162ea"
format_stamp: "Formatted at 2026-09-28 14:58:07 on dist-test-slave-sf70"
server_key: "f4a36bc45c6030ea9efc3255ade9c93b"
server_key_iv: "85ba25c6b69da86501611d930411ec01"
server_key_version: "encryptionkey@0"
I20260928 14:58:07.703310 4446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:07.726434 4446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:07.728737 4446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:07.728899 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:07.733007 4446 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:37403
I20260928 14:58:07.733042 4513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:37403 every 8 connection(s)
I20260928 14:58:07.733420 4446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:07.734256 4514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:07.736581 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea: Bootstrap starting.
I20260928 14:58:07.737144 4514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:07.737440 4514 log.cc:824] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:07.738093 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea: No bootstrap required, opened a new log
I20260928 14:58:07.739669 4514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e86bdcf5584312a7761b3ba4f162ea" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37403 } }
I20260928 14:58:07.739806 4514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:07.739862 4514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e86bdcf5584312a7761b3ba4f162ea, State: Initialized, Role: FOLLOWER
I20260928 14:58:07.740036 4514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e86bdcf5584312a7761b3ba4f162ea" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37403 } }
I20260928 14:58:07.740099 4514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:07.740139 4514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:07.740186 4514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:07.740703 4514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e86bdcf5584312a7761b3ba4f162ea" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37403 } }
I20260928 14:58:07.740809 4514 leader_election.cc:302] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40e86bdcf5584312a7761b3ba4f162ea; no voters:
I20260928 14:58:07.740988 4514 leader_election.cc:288] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:07.741011 4517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:07.741245 4514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:07.741389 4517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [term 1 LEADER]: Becoming Leader. State: Replica: 40e86bdcf5584312a7761b3ba4f162ea, State: Running, Role: LEADER
I20260928 14:58:07.741394 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 4446
I20260928 14:58:07.741513 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:07.741500 4517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e86bdcf5584312a7761b3ba4f162ea" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37403 } }
I20260928 14:58:07.741691 2035 external_mini_cluster.cc:1428] Setting key de8941ee764a1ac0b4d6187f87c3e311
I20260928 14:58:07.741879 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e86bdcf5584312a7761b3ba4f162ea. Latest consensus state: current_term: 1 leader_uuid: "40e86bdcf5584312a7761b3ba4f162ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e86bdcf5584312a7761b3ba4f162ea" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37403 } } }
I20260928 14:58:07.741968 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:07.742202 4526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:07.741890 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40e86bdcf5584312a7761b3ba4f162ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e86bdcf5584312a7761b3ba4f162ea" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37403 } } }
I20260928 14:58:07.742452 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:07.742731 4526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:07.744556 4526 catalog_manager.cc:1347] Generated new cluster ID: a8dc693a4ab44ca38975c536b580f681
I20260928 14:58:07.744612 4526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:07 dist-test-slave-sf70 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:07.749857 4526 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:07.750514 4526 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:07.753407 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:07.742188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.1.252.254:37403)
0928 14:58:07.742590 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:07.742598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:07.742630 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:58:07.742741 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:07.742747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:07.742818 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:07.742970 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:07.743821 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:07.745210 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:07.746242 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:07.746891 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:07.747001 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:07.749890 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:07.749893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:07.749894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:07.749906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:07.751828 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:07.752237 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:07.752240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:07.752240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:07.752338 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:07.752556 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:07.752557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:07.752557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:07.752788 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:07.752970 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:07.753069 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:58:07.753235 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":76,"threads_started":1}
I20260928 14:58:07.762296 4526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40e86bdcf5584312a7761b3ba4f162ea: Generated new TSK 0
I20260928 14:58:07.762511 4526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:07.791626 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37403
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:07.899816 4540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:07.900086 4540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:07.900112 4540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:07.901443 4540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:07.901487 4540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:07.901516 4540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:07.903160 4540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:07.903434 4540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:07.903703 4540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:07.909191 4546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:07.909348 4545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:07.909196 4548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:07.909511 4540 server_base.cc:1034] running on GCE node
I20260928 14:58:07.909703 4540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:07.909915 4540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:07.911051 4540 hybrid_clock.cc:648] HybridClock initialized: now 1790607487911040 us; error 27 us; skew 500 ppm
Sep 28 14:58:07 dist-test-slave-sf70 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:07.913542 4540 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:07.914069 4540 webserver.cc:460] Webserver started at http://127.1.252.193:43609/ using document root <none> and password file <none>
I20260928 14:58:07.914337 4540 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:07.914425 4540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:07.914585 4540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:07.915745 4540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "3fced349d57c471c8447dc0bfc8edd9f"
format_stamp: "Formatted at 2026-09-28 14:58:07 on dist-test-slave-sf70"
server_key: "fd97fc3b4a660d5102548d651d0582c4"
server_key_iv: "a74b416f4a897c5b24c907c598740f3a"
server_key_version: "encryptionkey@0"
I20260928 14:58:07.916146 4540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "3fced349d57c471c8447dc0bfc8edd9f"
format_stamp: "Formatted at 2026-09-28 14:58:07 on dist-test-slave-sf70"
server_key: "fd97fc3b4a660d5102548d651d0582c4"
server_key_iv: "a74b416f4a897c5b24c907c598740f3a"
server_key_version: "encryptionkey@0"
I20260928 14:58:07.917569 4540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:07.918435 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:07.918648 4540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:07.918767 4540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "3fced349d57c471c8447dc0bfc8edd9f"
format_stamp: "Formatted at 2026-09-28 14:58:07 on dist-test-slave-sf70"
server_key: "fd97fc3b4a660d5102548d651d0582c4"
server_key_iv: "a74b416f4a897c5b24c907c598740f3a"
server_key_version: "encryptionkey@0"
I20260928 14:58:07.918844 4540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:07.943675 4540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:07.945516 4540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:07.945680 4540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:07.945912 4540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:07.946219 4540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:07.946275 4540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:07.946314 4540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:07.946343 4540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:07.952057 4540 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:41617
I20260928 14:58:07.952153 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:41617 every 8 connection(s)
I20260928 14:58:07.952455 4540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:07.957257 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 4540
I20260928 14:58:07.957379 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:07.957612 2035 external_mini_cluster.cc:1428] Setting key d7bdd611604c277b287ea74f372fa8ee
Sep 28 14:58:07 dist-test-slave-sf70 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:07.962327 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:07.953174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:34437 (local address 127.1.252.254:37403)
0928 14:58:07.953230 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:07.953232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:07.953487 (+ 255us) server_negotiation.cc:407] Connection header received
0928 14:58:07.954033 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:07.954034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:07.954048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:07.954103 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:07.955055 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:07.955575 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:07.956675 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:07.956887 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:07.956975 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:07.959177 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:07.959179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:07.959179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:07.959188 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:07.960861 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:07.961252 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:07.961254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:07.961254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:07.961322 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:07.961561 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:07.961562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:07.961562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:07.961721 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:07.961796 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:07.962113 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 14:58:07.962217 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:58:07.962349 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:07.953144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37403 (local address 127.1.252.193:34437)
0928 14:58:07.953329 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:07.953388 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:07.953907 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:07.954189 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:07.954193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:07.954756 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:07.954944 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:07.954949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:07.955676 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:07.955677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:07.956574 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:07.956577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:07.956985 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:07.956986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:07.957129 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:07.957682 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:07.957692 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:07.959078 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:07.960962 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:07.960968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:07.960970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:07.961177 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:07.961392 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:07.961392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:07.961393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:07.961493 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:07.961792 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:07.961797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:07.961951 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 14:58:07.962056 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:58:07.962177 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260928 14:58:07.962980 4671 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37403
I20260928 14:58:07.963064 4671 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:07.963258 4671 heartbeater.cc:511] Master 127.1.252.254:37403 requested a full tablet report, sending...
I20260928 14:58:07.963855 4478 ts_manager.cc:194] Registered new tserver with Master: 3fced349d57c471c8447dc0bfc8edd9f (127.1.252.193:41617)
I20260928 14:58:07.964843 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:34437
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:07.983680 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37403
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:08.063616 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:08.063813 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:08.063836 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:08.065037 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:08.065081 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:08.065109 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:08.066473 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:08.066666 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:08.066835 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:08.071946 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:08.072048 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:08.072052 4676 server_base.cc:1034] running on GCE node
W20260928 14:58:08.072237 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:08.072474 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:08.072667 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:08.073803 4676 hybrid_clock.cc:648] HybridClock initialized: now 1790607488073772 us; error 46 us; skew 500 ppm
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:08.076241 4676 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:08.076647 4676 webserver.cc:460] Webserver started at http://127.1.252.194:35959/ using document root <none> and password file <none>
I20260928 14:58:08.076875 4676 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:08.076920 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:08.077049 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:08.078105 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "029fa97ce4b14c2abfd441eace1cdc60"
format_stamp: "Formatted at 2026-09-28 14:58:08 on dist-test-slave-sf70"
server_key: "b19fb3932d576ba183fff53d0cf99e21"
server_key_iv: "a845c2e374dfab8ef7470a14bbd20a26"
server_key_version: "encryptionkey@0"
I20260928 14:58:08.078457 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "029fa97ce4b14c2abfd441eace1cdc60"
format_stamp: "Formatted at 2026-09-28 14:58:08 on dist-test-slave-sf70"
server_key: "b19fb3932d576ba183fff53d0cf99e21"
server_key_iv: "a845c2e374dfab8ef7470a14bbd20a26"
server_key_version: "encryptionkey@0"
I20260928 14:58:08.079720 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:58:08.080490 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.080663 4676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.080750 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "029fa97ce4b14c2abfd441eace1cdc60"
format_stamp: "Formatted at 2026-09-28 14:58:08 on dist-test-slave-sf70"
server_key: "b19fb3932d576ba183fff53d0cf99e21"
server_key_iv: "a845c2e374dfab8ef7470a14bbd20a26"
server_key_version: "encryptionkey@0"
I20260928 14:58:08.080823 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:08.091450 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:08.093587 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:08.093744 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:08.093958 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:08.094251 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:08.094285 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.094305 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:08.094353 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.100085 4676 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:43369
I20260928 14:58:08.100256 4805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:43369 every 8 connection(s)
I20260928 14:58:08.100486 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:08.108484 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 4676
I20260928 14:58:08.108597 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:08.108865 2035 external_mini_cluster.cc:1428] Setting key 9bb599b9077d418ba9d5df1726d3b40b
I20260928 14:58:08.110481 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:08.101159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37403 (local address 127.1.252.194:40303)
0928 14:58:08.101336 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:08.101381 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:08.101927 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:08.102175 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:08.102179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:08.102738 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:08.102920 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:08.102925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.103626 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.103627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:08.104484 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:08.104486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.104873 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.104874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:08.105002 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:08.105641 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:08.105650 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:08.107111 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:08.108906 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:08.108913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:08.108916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:08.109264 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:08.109451 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:08.109452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:08.109453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:08.109605 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:08.109943 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:08.109950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:08.110082 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 14:58:08.110183 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:58:08.110308 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260928 14:58:08.110503 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:08.101190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:40303 (local address 127.1.252.254:37403)
0928 14:58:08.101284 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:08.101287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:08.101458 (+ 171us) server_negotiation.cc:407] Connection header received
0928 14:58:08.102037 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:08.102038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:08.102050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:08.102105 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:08.103018 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.103535 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.104572 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.104780 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.104863 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:08.107193 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:08.107194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:08.107194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:08.107200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:08.108821 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:08.109325 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:08.109326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:08.109326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:08.109382 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:08.109681 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:08.109682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:08.109682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:08.109842 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:08.109927 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:08.110234 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 14:58:08.110348 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:58:08.111039 4806 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37403
I20260928 14:58:08.111176 4806 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:08.111397 4806 heartbeater.cc:511] Master 127.1.252.254:37403 requested a full tablet report, sending...
I20260928 14:58:08.111810 4478 ts_manager.cc:194] Registered new tserver with Master: 029fa97ce4b14c2abfd441eace1cdc60 (127.1.252.194:43369)
I20260928 14:58:08.112365 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:40303
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:08.136009 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37403
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:08.215512 4811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:08.215737 4811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:08.215772 4811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:08.216967 4811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:08.217036 4811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:08.217082 4811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:08.218505 4811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:08.218760 4811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:08.218990 4811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:08.224044 4819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:08.224071 4817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:08.224085 4816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:08.224298 4811 server_base.cc:1034] running on GCE node
I20260928 14:58:08.224490 4811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:08.224717 4811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:08.225899 4811 hybrid_clock.cc:648] HybridClock initialized: now 1790607488225875 us; error 33 us; skew 500 ppm
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:08.228418 4811 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:08.228953 4811 webserver.cc:460] Webserver started at http://127.1.252.195:39119/ using document root <none> and password file <none>
I20260928 14:58:08.229197 4811 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:08.229274 4811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:08.229425 4811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:08.230531 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "a3b21183380a4ff9be301d5b4ccd0c1a"
format_stamp: "Formatted at 2026-09-28 14:58:08 on dist-test-slave-sf70"
server_key: "d9c171604790be868d631eb4dce22676"
server_key_iv: "6bf1d035cdfe9d5c464e303ce60f0b72"
server_key_version: "encryptionkey@0"
I20260928 14:58:08.230898 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "a3b21183380a4ff9be301d5b4ccd0c1a"
format_stamp: "Formatted at 2026-09-28 14:58:08 on dist-test-slave-sf70"
server_key: "d9c171604790be868d631eb4dce22676"
server_key_iv: "6bf1d035cdfe9d5c464e303ce60f0b72"
server_key_version: "encryptionkey@0"
I20260928 14:58:08.232259 4811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:08.232954 4826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.233127 4811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:08.233214 4811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "a3b21183380a4ff9be301d5b4ccd0c1a"
format_stamp: "Formatted at 2026-09-28 14:58:08 on dist-test-slave-sf70"
server_key: "d9c171604790be868d631eb4dce22676"
server_key_iv: "6bf1d035cdfe9d5c464e303ce60f0b72"
server_key_version: "encryptionkey@0"
I20260928 14:58:08.233299 4811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:08.244362 4811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:08.246546 4811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:08.246703 4811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:08.246927 4811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:08.247205 4811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:08.247262 4811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.247301 4811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:08.247329 4811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:08.253284 4811 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:38185
I20260928 14:58:08.253377 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:38185 every 8 connection(s)
I20260928 14:58:08.253676 4811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:08.260469 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 4811
I20260928 14:58:08.260568 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:08.260784 2035 external_mini_cluster.cc:1428] Setting key f3eb5b4a6dba94aca749349ef6c80c5c
I20260928 14:58:08.263020 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:08.254399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:54155 (local address 127.1.252.254:37403)
0928 14:58:08.254466 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:08.254467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:08.254684 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:58:08.255200 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:08.255201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:08.255215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:08.255279 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:08.256263 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.256787 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.257775 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.257987 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.258071 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:08.260151 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:08.260153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:08.260153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:08.260161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:08.261694 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:08.262041 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:08.262042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:08.262042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:08.262129 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:08.262369 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:08.262369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:08.262370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:08.262530 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:08.262582 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:08.262890 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 14:58:08.262958 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:58:08.263195 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:08.254392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37403 (local address 127.1.252.195:54155)
0928 14:58:08.254571 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:08.254605 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:08.255092 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:08.255346 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:08.255350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:08.255951 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:08.256150 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:08.256155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.256884 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.256885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:08.257660 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:08.257662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:08.258080 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:08.258080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:08.258200 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:08.258686 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:08.258694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:08.260059 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:08.261770 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:08.261775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:08.261776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:08.261977 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:08.262200 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:08.262200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:08.262200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:08.262291 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:08.262584 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:08.262589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:08.262714 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 14:58:08.262870 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:58:08.262998 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260928 14:58:08.263672 4941 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37403
I20260928 14:58:08.263752 4941 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:08.263991 4941 heartbeater.cc:511] Master 127.1.252.254:37403 requested a full tablet report, sending...
I20260928 14:58:08.264387 4478 ts_manager.cc:194] Registered new tserver with Master: a3b21183380a4ff9be301d5b4ccd0c1a (127.1.252.195:38185)
I20260928 14:58:08.264715 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:54155
I20260928 14:58:08.272536 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:58:08.283097 4528 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:08.282295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.1.252.254:37403)
0928 14:58:08.282360 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:08.282362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:08.282481 (+ 119us) server_negotiation.cc:407] Connection header received
0928 14:58:08.282522 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:08.282522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:08.282531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:08.282583 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:08.283047 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36144: BlockingRecv error: recv got EOF from 127.0.0.1:36144 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:58:08.284189 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 4540
I20260928 14:58:08.289304 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 4676
I20260928 14:58:08.293583 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 4811
I20260928 14:58:08.298324 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 4446
2026-09-28T14:58:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1853 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4953](info): set up 2 sockets
Sep 28 14:58:08 dist-test-slave-sf70 krb5kdc[4953](info): commencing operation
krb5kdc: starting...
W20260928 14:58:09.349503 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:09 dist-test-slave-sf70 krb5kdc[4953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:09.468037 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:34837
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:09.551149 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:09.551414 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:09.551456 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:09.552728 4966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:09.552802 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:09.552829 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:09.552855 4966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:09.552886 4966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:09.554402 4966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:34837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:09.554677 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:09.554932 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:09.559778 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:09.559769 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:09.559787 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:09.560032 4966 server_base.cc:1034] running on GCE node
I20260928 14:58:09.560370 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:09.560648 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:09.561790 4966 hybrid_clock.cc:648] HybridClock initialized: now 1790607489561771 us; error 35 us; skew 500 ppm
Sep 28 14:58:09 dist-test-slave-sf70 krb5kdc[4953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:09.570777 4966 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:09.571269 4966 webserver.cc:460] Webserver started at http://127.1.252.254:33301/ using document root <none> and password file <none>
I20260928 14:58:09.571522 4966 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:09.571594 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:09.571749 4966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:09.572881 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "9774bf113d7b4d18a764773fe8e0bf76"
format_stamp: "Formatted at 2026-09-28 14:58:09 on dist-test-slave-sf70"
server_key: "2808dc1a5012df1d785ba6c6485683f7"
server_key_iv: "38aad6dbd96fe9a2404cf712f2af7022"
server_key_version: "encryptionkey@0"
I20260928 14:58:09.573251 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "9774bf113d7b4d18a764773fe8e0bf76"
format_stamp: "Formatted at 2026-09-28 14:58:09 on dist-test-slave-sf70"
server_key: "2808dc1a5012df1d785ba6c6485683f7"
server_key_iv: "38aad6dbd96fe9a2404cf712f2af7022"
server_key_version: "encryptionkey@0"
I20260928 14:58:09.574554 4966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:58:09.575300 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:09.575480 4966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:09.575578 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "9774bf113d7b4d18a764773fe8e0bf76"
format_stamp: "Formatted at 2026-09-28 14:58:09 on dist-test-slave-sf70"
server_key: "2808dc1a5012df1d785ba6c6485683f7"
server_key_iv: "38aad6dbd96fe9a2404cf712f2af7022"
server_key_version: "encryptionkey@0"
I20260928 14:58:09.575661 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:09.583658 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:09.585392 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:09.585561 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:09.589702 4966 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:34837
I20260928 14:58:09.589735 5033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:34837 every 8 connection(s)
I20260928 14:58:09.590072 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:09.590876 5034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:09.592854 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 4966
I20260928 14:58:09.592962 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:09.593142 2035 external_mini_cluster.cc:1428] Setting key 0222f6307a38f53752718cec627ca9dd
I20260928 14:58:09.593503 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76: Bootstrap starting.
I20260928 14:58:09.594146 5034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:09.594450 5034 log.cc:824] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:09.595219 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76: No bootstrap required, opened a new log
I20260928 14:58:09.597365 5034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774bf113d7b4d18a764773fe8e0bf76" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34837 } }
I20260928 14:58:09.597530 5034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:09.597555 5034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9774bf113d7b4d18a764773fe8e0bf76, State: Initialized, Role: FOLLOWER
I20260928 14:58:09.597698 5034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774bf113d7b4d18a764773fe8e0bf76" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34837 } }
I20260928 14:58:09.597770 5034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:09.597792 5034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:09.597857 5034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:09.598575 5034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774bf113d7b4d18a764773fe8e0bf76" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34837 } }
I20260928 14:58:09.598690 5034 leader_election.cc:302] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9774bf113d7b4d18a764773fe8e0bf76; no voters:
I20260928 14:58:09.598860 5034 leader_election.cc:288] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:09.598929 5039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:09.599063 5039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [term 1 LEADER]: Becoming Leader. State: Replica: 9774bf113d7b4d18a764773fe8e0bf76, State: Running, Role: LEADER
I20260928 14:58:09.599138 5034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:09.599190 5039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774bf113d7b4d18a764773fe8e0bf76" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34837 } }
Sep 28 14:58:09 dist-test-slave-sf70 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:09.600293 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9774bf113d7b4d18a764773fe8e0bf76. Latest consensus state: current_term: 1 leader_uuid: "9774bf113d7b4d18a764773fe8e0bf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774bf113d7b4d18a764773fe8e0bf76" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34837 } } }
I20260928 14:58:09.600384 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:09.600703 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9774bf113d7b4d18a764773fe8e0bf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9774bf113d7b4d18a764773fe8e0bf76" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34837 } } }
I20260928 14:58:09.600776 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:09.600972 5050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:09.601568 5050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:09.603183 5050 catalog_manager.cc:1347] Generated new cluster ID: d5ac638c4a544680ac6dc591f73331fb
I20260928 14:58:09.603231 5050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:09.604460 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:09.593663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.1.252.254:34837)
0928 14:58:09.593843 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:09.593850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:09.593880 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:58:09.594327 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:09.594333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:09.594385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:09.594496 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:09.595324 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:09.596261 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:09.597407 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:09.598003 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:09.598082 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:09.600949 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:09.600953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:09.600953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:09.600965 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:09.602860 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:09.603256 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:09.603259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:09.603259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:09.603374 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:09.603602 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:09.603602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:09.603603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:09.603882 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:09.604043 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:09.604149 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:58:09.604268 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260928 14:58:09.616829 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:58:09.622015 5050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:09.622807 5050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:09.631771 5050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9774bf113d7b4d18a764773fe8e0bf76: Generated new TSK 0
I20260928 14:58:09.631974 5050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:09.634303 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:09.630547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.1.252.254:34837)
0928 14:58:09.630594 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:09.630596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:09.630699 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:58:09.630761 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:09.630761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:09.630773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:09.630829 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:09.631253 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:09.632025 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:09.632766 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:09.633121 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:09.633194 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:09.633849 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:09.633850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:09.633851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:09.634070 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:09.634089 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:09.634095 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:09.634157 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:09.634204 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 14:58:09.634228 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:58:09.636771 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 4966
2026-09-28T14:58:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1336 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:09 dist-test-slave-sf70 krb5kdc[5068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:09 dist-test-slave-sf70 krb5kdc[5068](info): set up 2 sockets
Sep 28 14:58:09 dist-test-slave-sf70 krb5kdc[5068](info): commencing operation
krb5kdc: starting...
W20260928 14:58:10.657629 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:10 dist-test-slave-sf70 krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:10.775727 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:44867
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:10.855505 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:10.855710 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:10.855731 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:10.856828 5081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:10.856868 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:10.856881 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:10.856894 5081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:10.856907 5081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:10.858304 5081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:10.858525 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:10.858693 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:10.863457 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:10.863426 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:10.863628 5081 server_base.cc:1034] running on GCE node
W20260928 14:58:10.863718 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:10.864028 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:10.864326 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:10.865468 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790607490865454 us; error 29 us; skew 500 ppm
Sep 28 14:58:10 dist-test-slave-sf70 krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:10.867980 5081 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:10.868407 5081 webserver.cc:460] Webserver started at http://127.1.252.254:41795/ using document root <none> and password file <none>
I20260928 14:58:10.868641 5081 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:10.868709 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:10.868861 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:10.869947 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "412292baad2a41898e17110a4b740679"
format_stamp: "Formatted at 2026-09-28 14:58:10 on dist-test-slave-sf70"
server_key: "7ca611a3f5207cd041b664d00b264873"
server_key_iv: "b2230e3a4510efccc2c066dc60bd94e6"
server_key_version: "encryptionkey@0"
I20260928 14:58:10.870316 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "412292baad2a41898e17110a4b740679"
format_stamp: "Formatted at 2026-09-28 14:58:10 on dist-test-slave-sf70"
server_key: "7ca611a3f5207cd041b664d00b264873"
server_key_iv: "b2230e3a4510efccc2c066dc60bd94e6"
server_key_version: "encryptionkey@0"
I20260928 14:58:10.871613 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:10.872426 5096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:10.872617 5081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:10.872725 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "412292baad2a41898e17110a4b740679"
format_stamp: "Formatted at 2026-09-28 14:58:10 on dist-test-slave-sf70"
server_key: "7ca611a3f5207cd041b664d00b264873"
server_key_iv: "b2230e3a4510efccc2c066dc60bd94e6"
server_key_version: "encryptionkey@0"
I20260928 14:58:10.872822 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:10.887475 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:10.889810 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:10.889981 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:10.893843 5081 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:44867
I20260928 14:58:10.893878 5148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:44867 every 8 connection(s)
I20260928 14:58:10.894229 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:10.895174 5149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:10.897378 5149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679: Bootstrap starting.
I20260928 14:58:10.897881 5149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:10.898097 5149 log.cc:824] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:10.898715 5149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679: No bootstrap required, opened a new log
I20260928 14:58:10.899837 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5081
I20260928 14:58:10.899982 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:10.900161 2035 external_mini_cluster.cc:1428] Setting key 568c3b89df0a56fa6b9c4efa210c6259
I20260928 14:58:10.900512 5149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412292baad2a41898e17110a4b740679" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44867 } }
I20260928 14:58:10.900689 5149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:10.900753 5149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412292baad2a41898e17110a4b740679, State: Initialized, Role: FOLLOWER
I20260928 14:58:10.900926 5149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412292baad2a41898e17110a4b740679" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44867 } }
I20260928 14:58:10.901011 5149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:10.901042 5149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:10.901087 5149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:10.902014 5149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412292baad2a41898e17110a4b740679" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44867 } }
I20260928 14:58:10.902117 5149 leader_election.cc:302] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412292baad2a41898e17110a4b740679; no voters:
I20260928 14:58:10.902266 5149 leader_election.cc:288] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:10.902333 5154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:10.902522 5154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [term 1 LEADER]: Becoming Leader. State: Replica: 412292baad2a41898e17110a4b740679, State: Running, Role: LEADER
I20260928 14:58:10.902508 5149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:10.902667 5154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412292baad2a41898e17110a4b740679" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44867 } }
I20260928 14:58:10.903036 5154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412292baad2a41898e17110a4b740679. Latest consensus state: current_term: 1 leader_uuid: "412292baad2a41898e17110a4b740679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412292baad2a41898e17110a4b740679" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44867 } } }
I20260928 14:58:10.903182 5155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412292baad2a41898e17110a4b740679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412292baad2a41898e17110a4b740679" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44867 } } }
I20260928 14:58:10.903264 5154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [sys.catalog]: This master's current role is: LEADER
W20260928 14:58:10.904353 5169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:58:10.904575 5169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:58:10.904441 5155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:10.904749 5171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:10.904903 5171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:10.906399 5171 catalog_manager.cc:1347] Generated new cluster ID: 7b39b4ea55a143528cebcbea8567d0ba
I20260928 14:58:10.906459 5171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:10 dist-test-slave-sf70 krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:10.911243 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:10.900668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.1.252.254:44867)
0928 14:58:10.900849 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:10.900858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:10.900897 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:58:10.901008 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:10.901015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:10.901070 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:10.901210 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:10.901784 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:10.903234 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:10.904182 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:10.904772 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:10.904907 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:10.907762 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:10.907766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:10.907766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:10.907777 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:10.909663 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:10.910068 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:10.910070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:10.910071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:10.910177 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:10.910403 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:10.910404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:10.910404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:10.910672 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:10.910820 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:10.910933 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 14:58:10.911057 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 14:58:10.920080 5171 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:10.920560 5171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:10.929529 5171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412292baad2a41898e17110a4b740679: Generated new TSK 0
I20260928 14:58:10.929720 5171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:10.960908 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44867
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:11.041633 5175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:11.041852 5175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:11.041879 5175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:11.043157 5175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:11.043202 5175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:11.043267 5175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:11.044912 5175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:11.045168 5175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:11.045425 5175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:11.050773 5183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:11.050894 5180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:11.050825 5181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:11.050940 5175 server_base.cc:1034] running on GCE node
I20260928 14:58:11.051157 5175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:11.051352 5175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:11.052488 5175 hybrid_clock.cc:648] HybridClock initialized: now 1790607491052474 us; error 30 us; skew 500 ppm
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:11.055018 5175 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:11.055514 5175 webserver.cc:460] Webserver started at http://127.1.252.193:39613/ using document root <none> and password file <none>
I20260928 14:58:11.055756 5175 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:11.055821 5175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:11.055989 5175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:11.057003 5175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "65cc0aee600145e7b1a8b1d9d8980abc"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "d190e5d3cb53945fa8dfce6a97e19efc"
server_key_iv: "ffd7ff927d0d55906259ee90c378026c"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.057363 5175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "65cc0aee600145e7b1a8b1d9d8980abc"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "d190e5d3cb53945fa8dfce6a97e19efc"
server_key_iv: "ffd7ff927d0d55906259ee90c378026c"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.058697 5175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:11.059685 5190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.059868 5175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:11.059963 5175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "65cc0aee600145e7b1a8b1d9d8980abc"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "d190e5d3cb53945fa8dfce6a97e19efc"
server_key_iv: "ffd7ff927d0d55906259ee90c378026c"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.060045 5175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:11.071452 5175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:11.073325 5175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:11.073490 5175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:11.073714 5175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:11.074018 5175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:11.074070 5175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.074110 5175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:11.074137 5175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.079833 5175 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:43229
I20260928 14:58:11.079946 5304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:43229 every 8 connection(s)
I20260928 14:58:11.080258 5175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:11.085510 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5175
I20260928 14:58:11.085618 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:11.085825 2035 external_mini_cluster.cc:1428] Setting key fbbacff9e179be7582f5e440bdcbb4d6
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:11.089439 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.080946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43651 (local address 127.1.252.254:44867)
0928 14:58:11.081020 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:11.081023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:11.081203 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:58:11.081651 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:11.081652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:11.081666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:11.081717 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:11.082593 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.083096 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.084100 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.084289 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.084343 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.086589 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:11.086590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:11.086591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:11.086598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:11.088078 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.088489 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.088491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.088491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.088549 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.088799 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.088800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.088800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.088915 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:11.088985 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:11.089261 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 14:58:11.089351 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:58:11.089620 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.080914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44867 (local address 127.1.252.193:43651)
0928 14:58:11.081090 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:11.081124 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:11.081571 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:11.081767 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:11.081770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:11.082320 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:11.082515 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:11.082520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.083173 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.083173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:11.084032 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:11.084035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.084361 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.084361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:11.084459 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.084854 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:11.084862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:11.086483 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:11.088171 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:11.088178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:11.088180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:11.088399 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:11.088615 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:11.088615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:11.088616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:11.088727 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:11.088987 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:11.088994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:11.089125 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 14:58:11.089223 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:58:11.089381 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260928 14:58:11.090088 5305 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44867
I20260928 14:58:11.090188 5305 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:11.090350 5305 heartbeater.cc:511] Master 127.1.252.254:44867 requested a full tablet report, sending...
I20260928 14:58:11.090894 5113 ts_manager.cc:194] Registered new tserver with Master: 65cc0aee600145e7b1a8b1d9d8980abc (127.1.252.193:43229)
I20260928 14:58:11.091495 5113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:43651
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:11.110994 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44867
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:11.190079 5310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:11.190270 5310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:11.190294 5310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:11.191421 5310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:11.191463 5310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:11.191490 5310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:11.192937 5310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:11.193171 5310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:11.193390 5310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:11.198503 5316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:11.198549 5310 server_base.cc:1034] running on GCE node
W20260928 14:58:11.198513 5318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:11.198520 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:11.198917 5310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:11.199160 5310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:11.200322 5310 hybrid_clock.cc:648] HybridClock initialized: now 1790607491200309 us; error 30 us; skew 500 ppm
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:11.202817 5310 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:11.203176 5310 webserver.cc:460] Webserver started at http://127.1.252.194:35405/ using document root <none> and password file <none>
I20260928 14:58:11.203390 5310 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:11.203433 5310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:11.203524 5310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:11.204706 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "24896444c5c1495f8ff80e34e2fe62c1"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "1431721f4f0fe513a5a1d5a20e7ccad1"
server_key_iv: "138eddf2ab512807122e92f370bec4d7"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.205106 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "24896444c5c1495f8ff80e34e2fe62c1"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "1431721f4f0fe513a5a1d5a20e7ccad1"
server_key_iv: "138eddf2ab512807122e92f370bec4d7"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.206367 5310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:11.207098 5325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.207274 5310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:11.207365 5310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "24896444c5c1495f8ff80e34e2fe62c1"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "1431721f4f0fe513a5a1d5a20e7ccad1"
server_key_iv: "138eddf2ab512807122e92f370bec4d7"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.207521 5310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:11.223860 5310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:11.225942 5310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:11.226081 5310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:11.226300 5310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:11.226606 5310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:11.226639 5310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.226660 5310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:11.226704 5310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.232487 5310 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:39475
I20260928 14:58:11.232532 5438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:39475 every 8 connection(s)
I20260928 14:58:11.232815 5310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:11.235321 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5310
I20260928 14:58:11.235440 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:11.235622 2035 external_mini_cluster.cc:1428] Setting key 3e1b58356525cf398f8bff882456e0fb
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:11.242532 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.233485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:41401 (local address 127.1.252.254:44867)
0928 14:58:11.233551 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:11.233553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:11.233767 (+ 214us) server_negotiation.cc:407] Connection header received
0928 14:58:11.234313 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:11.234314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:11.234327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:11.234402 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:11.235315 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.236071 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.237180 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.237498 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.237585 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.239680 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:11.239681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:11.239681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:11.239689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:11.241093 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.241528 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.241530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.241530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.241596 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.241853 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.241854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.241854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.241987 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:11.242062 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:11.242338 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 14:58:11.242438 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:58:11.242745 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.233461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44867 (local address 127.1.252.194:41401)
0928 14:58:11.233648 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:11.233680 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:11.234200 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:11.234467 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:11.234471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:11.235026 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:11.235207 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:11.235211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.236160 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.236161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:11.237085 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:11.237087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.237608 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.237608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:11.237714 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.238226 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:11.238234 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:11.239583 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:11.241204 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:11.241211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:11.241213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:11.241447 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:11.241665 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:11.241666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:11.241666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:11.241782 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:11.242063 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:11.242070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:11.242208 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:58:11.242316 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:58:11.242505 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260928 14:58:11.243242 5439 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44867
I20260928 14:58:11.243319 5439 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:11.243528 5439 heartbeater.cc:511] Master 127.1.252.254:44867 requested a full tablet report, sending...
I20260928 14:58:11.243944 5113 ts_manager.cc:194] Registered new tserver with Master: 24896444c5c1495f8ff80e34e2fe62c1 (127.1.252.194:39475)
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
I20260928 14:58:11.244489 5113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:41401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:11.263459 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44867
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:11.343115 5444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:11.343300 5444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:11.343321 5444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:11.344385 5444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:11.344421 5444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:11.344446 5444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:11.345768 5444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:11.345940 5444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:11.346098 5444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:11.350847 5452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:11.350916 5444 server_base.cc:1034] running on GCE node
W20260928 14:58:11.350873 5449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:11.350872 5450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:11.351190 5444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:11.351440 5444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:11.352599 5444 hybrid_clock.cc:648] HybridClock initialized: now 1790607491352592 us; error 39 us; skew 500 ppm
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:11.354993 5444 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:11.355450 5444 webserver.cc:460] Webserver started at http://127.1.252.195:35597/ using document root <none> and password file <none>
I20260928 14:58:11.355680 5444 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:11.355747 5444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:11.355878 5444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:11.356982 5444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "6bc834aa2d174369aee61f09c87077b6"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "2d8c5dec2d395b3976ee79255316baab"
server_key_iv: "34741ed61aae7ada0b4cfdb4aeee5981"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.357328 5444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "6bc834aa2d174369aee61f09c87077b6"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "2d8c5dec2d395b3976ee79255316baab"
server_key_iv: "34741ed61aae7ada0b4cfdb4aeee5981"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.358603 5444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:58:11.359334 5459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.359500 5444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:11.359587 5444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "6bc834aa2d174369aee61f09c87077b6"
format_stamp: "Formatted at 2026-09-28 14:58:11 on dist-test-slave-sf70"
server_key: "2d8c5dec2d395b3976ee79255316baab"
server_key_iv: "34741ed61aae7ada0b4cfdb4aeee5981"
server_key_version: "encryptionkey@0"
I20260928 14:58:11.359666 5444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:11.370932 5444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:11.373210 5444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:11.373377 5444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:11.373606 5444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:11.373907 5444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:11.373960 5444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.374001 5444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:11.374028 5444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:11.380085 5444 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:40561
I20260928 14:58:11.380139 5572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:40561 every 8 connection(s)
I20260928 14:58:11.380479 5444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:11.388180 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5444
I20260928 14:58:11.388283 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:11.388497 2035 external_mini_cluster.cc:1428] Setting key 07a677c6071371135cc4530f793c9081
I20260928 14:58:11.389299 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.381182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:36493 (local address 127.1.252.254:44867)
0928 14:58:11.381229 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:11.381232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:11.381466 (+ 234us) server_negotiation.cc:407] Connection header received
0928 14:58:11.381913 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:11.381914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:11.381925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:11.381978 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:11.382819 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.383319 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.384356 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.384546 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.384600 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.386594 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:11.386595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:11.386595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:11.386602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:11.387753 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.388262 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.388263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.388264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.388334 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.388647 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.388648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.388649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.388772 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:11.388853 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:11.389131 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:58:11.389212 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:58:11.389477 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.381172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44867 (local address 127.1.252.195:36493)
0928 14:58:11.381339 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:11.381386 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:11.381832 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:11.382024 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:11.382028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:11.382564 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:11.382748 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:11.382752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.383400 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.383401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:11.384288 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:11.384290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.384617 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.384618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:11.384710 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.385214 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:11.385223 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:11.386510 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:11.387824 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:11.387829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:11.387831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:11.388176 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:11.388444 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:11.388444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:11.388444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:11.388575 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:11.388842 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:11.388849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:11.388983 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 14:58:11.389093 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:58:11.389243 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260928 14:58:11.389887 5573 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44867
I20260928 14:58:11.389967 5573 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:11.390152 5573 heartbeater.cc:511] Master 127.1.252.254:44867 requested a full tablet report, sending...
I20260928 14:58:11.390516 5113 ts_manager.cc:194] Registered new tserver with Master: 6bc834aa2d174369aee61f09c87077b6 (127.1.252.195:40561)
I20260928 14:58:11.390965 5113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:36493
I20260928 14:58:11.400321 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607491, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:11.422089 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:11.415494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.1.252.254:44867)
0928 14:58:11.415581 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:11.415584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:11.415637 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:58:11.415682 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:11.415682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:11.415690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:11.415738 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:11.416293 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.416770 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.417446 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:11.417629 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:11.417682 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:11.419794 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:11.419796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:11.419796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:11.419802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:11.421211 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.421502 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.421502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.421503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.421552 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:11.421692 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:11.421693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:11.421693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:11.421815 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:11.421903 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:11.421972 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:58:11.422029 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 14:58:11.422335 5113 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42896
I20260928 14:58:11.422732 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5175
I20260928 14:58:11.427120 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5310
I20260928 14:58:11.431396 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5444
I20260928 14:58:11.435264 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5081
2026-09-28T14:58:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1800 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5585](info): set up 2 sockets
Sep 28 14:58:11 dist-test-slave-sf70 krb5kdc[5585](info): commencing operation
krb5kdc: starting...
W20260928 14:58:12.577459 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:12 dist-test-slave-sf70 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:12.695827 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38007
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:12.775699 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:12.775974 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:12.776026 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:12.777174 5598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:12.777242 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:12.777266 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:12.777288 5598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:12.777316 5598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:12.778779 5598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:38007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:12.779055 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:12.779276 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:12.784031 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:12.784143 5598 server_base.cc:1034] running on GCE node
W20260928 14:58:12.784035 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:12.784035 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:12.784505 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:12.784737 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:12.785877 5598 hybrid_clock.cc:648] HybridClock initialized: now 1790607492785842 us; error 54 us; skew 500 ppm
Sep 28 14:58:12 dist-test-slave-sf70 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:12.788283 5598 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:12.788728 5598 webserver.cc:460] Webserver started at http://127.1.252.254:40229/ using document root <none> and password file <none>
I20260928 14:58:12.788952 5598 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:12.789023 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:12.789172 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:12.790251 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "72b8e3186d2947ff9f13d2e523890b77"
format_stamp: "Formatted at 2026-09-28 14:58:12 on dist-test-slave-sf70"
server_key: "3268c01d6c1565cefc63e4b6061d15c0"
server_key_iv: "705525b97417fb555fa44628e9e301bd"
server_key_version: "encryptionkey@0"
I20260928 14:58:12.790621 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "72b8e3186d2947ff9f13d2e523890b77"
format_stamp: "Formatted at 2026-09-28 14:58:12 on dist-test-slave-sf70"
server_key: "3268c01d6c1565cefc63e4b6061d15c0"
server_key_iv: "705525b97417fb555fa44628e9e301bd"
server_key_version: "encryptionkey@0"
I20260928 14:58:12.792011 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:58:12.792796 5613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:12.792973 5598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:12.793071 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "72b8e3186d2947ff9f13d2e523890b77"
format_stamp: "Formatted at 2026-09-28 14:58:12 on dist-test-slave-sf70"
server_key: "3268c01d6c1565cefc63e4b6061d15c0"
server_key_iv: "705525b97417fb555fa44628e9e301bd"
server_key_version: "encryptionkey@0"
I20260928 14:58:12.793159 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:12.826823 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:12.828811 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:12.828984 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:12.832939 5598 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:38007
I20260928 14:58:12.832978 5665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:38007 every 8 connection(s)
I20260928 14:58:12.833501 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:12.834146 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:12.836364 5666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77: Bootstrap starting.
I20260928 14:58:12.836966 5666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:12.837265 5666 log.cc:824] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:12.837968 5666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77: No bootstrap required, opened a new log
I20260928 14:58:12.839493 5666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8e3186d2947ff9f13d2e523890b77" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38007 } }
I20260928 14:58:12.839622 5666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:12.839645 5666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b8e3186d2947ff9f13d2e523890b77, State: Initialized, Role: FOLLOWER
I20260928 14:58:12.839818 5666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8e3186d2947ff9f13d2e523890b77" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38007 } }
I20260928 14:58:12.839879 5666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:12.839983 5666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:12.840034 5666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:12.840628 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5598
I20260928 14:58:12.840765 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:12.840961 2035 external_mini_cluster.cc:1428] Setting key 1842ea37463f4fe4d649ce9c2c373fea
I20260928 14:58:12.841152 5666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8e3186d2947ff9f13d2e523890b77" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38007 } }
I20260928 14:58:12.841243 5666 leader_election.cc:302] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b8e3186d2947ff9f13d2e523890b77; no voters:
I20260928 14:58:12.841439 5666 leader_election.cc:288] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:12.841544 5669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:12.841683 5666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:12.841733 5669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [term 1 LEADER]: Becoming Leader. State: Replica: 72b8e3186d2947ff9f13d2e523890b77, State: Running, Role: LEADER
I20260928 14:58:12.841826 5669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8e3186d2947ff9f13d2e523890b77" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38007 } }
I20260928 14:58:12.842134 5669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b8e3186d2947ff9f13d2e523890b77. Latest consensus state: current_term: 1 leader_uuid: "72b8e3186d2947ff9f13d2e523890b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8e3186d2947ff9f13d2e523890b77" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38007 } } }
I20260928 14:58:12.842219 5669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:12.842344 5671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b8e3186d2947ff9f13d2e523890b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8e3186d2947ff9f13d2e523890b77" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38007 } } }
I20260928 14:58:12.842442 5678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:12.842526 5671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:12.843243 5678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:12.844756 5678 catalog_manager.cc:1347] Generated new cluster ID: 2983660ecdca47f0ba5955e91c9f9f73
I20260928 14:58:12.844813 5678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:12 dist-test-slave-sf70 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:12.853606 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:12.841466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.1.252.254:38007)
0928 14:58:12.841745 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:12.841755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:12.841788 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:58:12.841901 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:12.841908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:12.841962 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:12.842107 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:12.843508 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:12.844917 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:12.845980 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:12.846590 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:12.846696 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:12.849410 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:12.849413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:12.849414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:12.849426 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:12.851974 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:12.852362 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:12.852365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:12.852365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:12.852497 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:12.852727 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:12.852728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:12.852728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:12.853000 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:12.853162 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:12.853279 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:58:12.853419 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260928 14:58:12.853837 5678 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:12.854385 5678 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:12.866086 5678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b8e3186d2947ff9f13d2e523890b77: Generated new TSK 0
I20260928 14:58:12.866273 5678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:12.896095 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38007
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:12.979039 5692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:12.979243 5692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:12.979266 5692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:12.980540 5692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:12.980597 5692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:12.980660 5692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:12.982220 5692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:12.982497 5692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:12.982738 5692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:12.987776 5697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:12.987756 5700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:12.987756 5698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:12.988059 5692 server_base.cc:1034] running on GCE node
I20260928 14:58:12.988344 5692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:12.988611 5692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:12.989778 5692 hybrid_clock.cc:648] HybridClock initialized: now 1790607492989757 us; error 36 us; skew 500 ppm
Sep 28 14:58:12 dist-test-slave-sf70 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:12.992396 5692 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:12.992821 5692 webserver.cc:460] Webserver started at http://127.1.252.193:32783/ using document root <none> and password file <none>
I20260928 14:58:12.993072 5692 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:12.993142 5692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:12.993287 5692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:12.994333 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "e18081b3e89a48e68fdc515a1fb77cb2"
format_stamp: "Formatted at 2026-09-28 14:58:12 on dist-test-slave-sf70"
server_key: "28144b5ceeee136cba636aa620aa8fbb"
server_key_iv: "501041b2542a5fad15eb7007c9416aa0"
server_key_version: "encryptionkey@0"
I20260928 14:58:12.994688 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "e18081b3e89a48e68fdc515a1fb77cb2"
format_stamp: "Formatted at 2026-09-28 14:58:12 on dist-test-slave-sf70"
server_key: "28144b5ceeee136cba636aa620aa8fbb"
server_key_iv: "501041b2542a5fad15eb7007c9416aa0"
server_key_version: "encryptionkey@0"
I20260928 14:58:12.996083 5692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:12.996929 5707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:12.997098 5692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:12.997197 5692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "e18081b3e89a48e68fdc515a1fb77cb2"
format_stamp: "Formatted at 2026-09-28 14:58:12 on dist-test-slave-sf70"
server_key: "28144b5ceeee136cba636aa620aa8fbb"
server_key_iv: "501041b2542a5fad15eb7007c9416aa0"
server_key_version: "encryptionkey@0"
I20260928 14:58:12.997279 5692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:13.013480 5692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:13.015470 5692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:13.015630 5692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:13.015856 5692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:13.016213 5692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:13.016268 5692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.016306 5692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:13.016335 5692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.022081 5692 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:41835
I20260928 14:58:13.022161 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:41835 every 8 connection(s)
I20260928 14:58:13.022486 5692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:13.031421 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5692
I20260928 14:58:13.031535 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:13.031701 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.023218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:45697 (local address 127.1.252.254:38007)
0928 14:58:13.023268 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:13.023270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:13.023524 (+ 254us) server_negotiation.cc:407] Connection header received
0928 14:58:13.024515 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:13.024516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:13.024531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:13.024585 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:13.025167 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.025671 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.026688 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.026886 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.026960 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.029151 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:13.029152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:13.029152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:13.029160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:13.030394 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.030753 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.030754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.030754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.030806 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.031020 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.031021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.031021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.031125 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:13.031176 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:13.031538 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 14:58:13.031636 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:58:13.031734 2035 external_mini_cluster.cc:1428] Setting key 023e6176c4c439469049408c0a80a591
I20260928 14:58:13.031941 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.023204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38007 (local address 127.1.252.193:45697)
0928 14:58:13.023391 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:13.023426 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:13.024419 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:13.024650 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:13.024654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:13.024903 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:13.025086 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:13.025090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.025779 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.025780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:13.026605 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:13.026607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.026981 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.026982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:13.027086 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.027544 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:13.027552 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:13.029056 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:13.030470 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:13.030476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:13.030478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:13.030688 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:13.030867 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:13.030868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:13.030868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:13.030963 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:13.031180 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:13.031184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:13.031367 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 14:58:13.031492 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:58:13.031669 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260928 14:58:13.032550 5821 heartbeater.cc:347] Connected to a master server at 127.1.252.254:38007
I20260928 14:58:13.032667 5821 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:13.032876 5821 heartbeater.cc:511] Master 127.1.252.254:38007 requested a full tablet report, sending...
I20260928 14:58:13.033394 5630 ts_manager.cc:194] Registered new tserver with Master: e18081b3e89a48e68fdc515a1fb77cb2 (127.1.252.193:41835)
I20260928 14:58:13.034267 5630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:45697
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:13.059783 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38007
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:13.139649 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:13.139875 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:13.139930 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:13.141055 5826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:13.141119 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:13.141168 5826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:13.142828 5826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:13.143077 5826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:13.143314 5826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:13.148319 5834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:13.148281 5832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:13.148284 5831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:13.148512 5826 server_base.cc:1034] running on GCE node
I20260928 14:58:13.148753 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:13.148988 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:13.150157 5826 hybrid_clock.cc:648] HybridClock initialized: now 1790607493150142 us; error 24 us; skew 500 ppm
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:13.152555 5826 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:13.152982 5826 webserver.cc:460] Webserver started at http://127.1.252.194:37723/ using document root <none> and password file <none>
I20260928 14:58:13.153169 5826 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:13.153214 5826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:13.153299 5826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:13.154294 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "27f104ae74f14513a265c060c0ed64b7"
format_stamp: "Formatted at 2026-09-28 14:58:13 on dist-test-slave-sf70"
server_key: "d6fd810e400ad83d99038dc4d0c85635"
server_key_iv: "17a2e103ea5338bf8bf57c9df496e9c5"
server_key_version: "encryptionkey@0"
I20260928 14:58:13.154580 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "27f104ae74f14513a265c060c0ed64b7"
format_stamp: "Formatted at 2026-09-28 14:58:13 on dist-test-slave-sf70"
server_key: "d6fd810e400ad83d99038dc4d0c85635"
server_key_iv: "17a2e103ea5338bf8bf57c9df496e9c5"
server_key_version: "encryptionkey@0"
I20260928 14:58:13.155845 5826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:13.156584 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.156755 5826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:13.156847 5826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "27f104ae74f14513a265c060c0ed64b7"
format_stamp: "Formatted at 2026-09-28 14:58:13 on dist-test-slave-sf70"
server_key: "d6fd810e400ad83d99038dc4d0c85635"
server_key_iv: "17a2e103ea5338bf8bf57c9df496e9c5"
server_key_version: "encryptionkey@0"
I20260928 14:58:13.156903 5826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:13.163007 5826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:13.164906 5826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:13.165011 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:13.165171 5826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:13.165460 5826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:13.165514 5826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.165555 5826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:13.165581 5826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.171120 5826 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:34207
I20260928 14:58:13.171202 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:34207 every 8 connection(s)
I20260928 14:58:13.171490 5826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:13.174312 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5826
I20260928 14:58:13.174425 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:13.174613 2035 external_mini_cluster.cc:1428] Setting key fcd7ab246a20f217b329a7eefae27c1f
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:13.181628 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.172263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:33571 (local address 127.1.252.254:38007)
0928 14:58:13.172309 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:13.172311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:13.172499 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:58:13.173442 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:13.173443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:13.173453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:13.173509 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:13.174083 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.174886 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.176069 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.176397 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.176497 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.178694 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:13.178696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:13.178696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:13.178705 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:13.180228 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.180661 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.180662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.180662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.180722 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.180981 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.180982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.180983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.181093 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:13.181171 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:13.181452 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 14:58:13.181540 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:13.181795 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.172219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38007 (local address 127.1.252.194:33571)
0928 14:58:13.172381 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:13.172418 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:13.173332 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:13.173561 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:13.173564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:13.173799 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:13.173989 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:13.173993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.174963 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.174964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:13.175982 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:13.175985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.176499 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.176499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:13.176622 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.177169 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:13.177177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:13.178587 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:13.180323 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:13.180329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:13.180331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:13.180553 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:13.180795 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:13.180795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:13.180796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:13.180906 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:13.181161 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:13.181167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:13.181312 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:58:13.181416 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:58:13.181567 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260928 14:58:13.182210 5955 heartbeater.cc:347] Connected to a master server at 127.1.252.254:38007
I20260928 14:58:13.182299 5955 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:13.182492 5955 heartbeater.cc:511] Master 127.1.252.254:38007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
I20260928 14:58:13.182888 5630 ts_manager.cc:194] Registered new tserver with Master: 27f104ae74f14513a265c060c0ed64b7 (127.1.252.194:34207)
I20260928 14:58:13.183281 5630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:33571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:13.202760 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38007
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:13.282967 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:13.283166 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:13.283190 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:13.284301 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:13.284337 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:13.284363 5960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:13.286398 5960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:13.286656 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:13.286895 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:13.291977 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:13.292065 5960 server_base.cc:1034] running on GCE node
W20260928 14:58:13.291985 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:13.291985 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:13.292436 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:13.292708 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:13.293913 5960 hybrid_clock.cc:648] HybridClock initialized: now 1790607493293842 us; error 89 us; skew 500 ppm
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:13.296548 5960 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:13.296983 5960 webserver.cc:460] Webserver started at http://127.1.252.195:45211/ using document root <none> and password file <none>
I20260928 14:58:13.297214 5960 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:13.297256 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:13.297417 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:13.298421 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "0f0e2c1e3bb742c8afc686e70c635d47"
format_stamp: "Formatted at 2026-09-28 14:58:13 on dist-test-slave-sf70"
server_key: "a11a2fe1bb82e70b136b4f109e64195c"
server_key_iv: "982f245824319e15125f0f9570b6674c"
server_key_version: "encryptionkey@0"
I20260928 14:58:13.298777 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "0f0e2c1e3bb742c8afc686e70c635d47"
format_stamp: "Formatted at 2026-09-28 14:58:13 on dist-test-slave-sf70"
server_key: "a11a2fe1bb82e70b136b4f109e64195c"
server_key_iv: "982f245824319e15125f0f9570b6674c"
server_key_version: "encryptionkey@0"
I20260928 14:58:13.300143 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:13.300990 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.301169 5960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:13.301255 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "0f0e2c1e3bb742c8afc686e70c635d47"
format_stamp: "Formatted at 2026-09-28 14:58:13 on dist-test-slave-sf70"
server_key: "a11a2fe1bb82e70b136b4f109e64195c"
server_key_iv: "982f245824319e15125f0f9570b6674c"
server_key_version: "encryptionkey@0"
I20260928 14:58:13.301329 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:13.316133 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:13.318040 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:13.318188 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:13.318432 5960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:13.318735 5960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:13.318768 5960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.318816 5960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:13.318845 5960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:13.324658 5960 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:41219
I20260928 14:58:13.324738 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:41219 every 8 connection(s)
I20260928 14:58:13.325038 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:13.327499 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 5960
I20260928 14:58:13.327611 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:13.327813 2035 external_mini_cluster.cc:1428] Setting key 8b3005cb91a8cd213941653ab44e3376
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:13.334280 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.325791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:51939 (local address 127.1.252.254:38007)
0928 14:58:13.325879 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:13.325882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:13.326041 (+ 159us) server_negotiation.cc:407] Connection header received
0928 14:58:13.326984 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:13.326986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:13.326997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:13.327055 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:13.327846 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.328692 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.329676 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.329933 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.330005 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.332115 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:13.332116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:13.332117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:13.332123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:13.333244 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.333609 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.333609 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.333610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.333658 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.333849 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.333849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.333850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.333937 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:13.333983 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:13.334164 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:58:13.334217 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:58:13.334539 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.325747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38007 (local address 127.1.252.195:51939)
0928 14:58:13.325928 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:13.325969 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:13.326872 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:13.327106 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:13.327109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:13.327438 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:13.327735 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:13.327743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.328787 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.328788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:13.329595 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:13.329597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.330016 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.330017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:13.330115 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.330631 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:13.330639 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:13.332038 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:13.333314 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:13.333320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:13.333321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:13.333544 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:13.333714 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:13.333715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:13.333715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:13.333808 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:13.333974 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:13.333976 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:13.334079 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 14:58:13.334154 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:58:13.334272 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260928 14:58:13.335134 6090 heartbeater.cc:347] Connected to a master server at 127.1.252.254:38007
I20260928 14:58:13.335246 6090 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:13.335420 6090 heartbeater.cc:511] Master 127.1.252.254:38007 requested a full tablet report, sending...
I20260928 14:58:13.335798 5630 ts_manager.cc:194] Registered new tserver with Master: 0f0e2c1e3bb742c8afc686e70c635d47 (127.1.252.195:41219)
I20260928 14:58:13.336187 5630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:51939
I20260928 14:58:13.339581 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:58:13.362793 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.355077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.1.252.193:41835)
0928 14:58:13.355247 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:13.355252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:13.355261 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:13.355286 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:13.355288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:13.355302 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:13.355387 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:13.355987 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.356719 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.357462 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.357772 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.357861 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.360107 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:13.360115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:13.360115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:13.360122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:13.361591 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.361946 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.361949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.361950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.362016 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.362212 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.362213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.362214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.362356 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:13.362522 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:13.362593 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:58:13.362673 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:13.370596 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:13.364119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.1.252.254:38007)
0928 14:58:13.364161 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:13.364163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:13.364229 (+ 66us) server_negotiation.cc:407] Connection header received
0928 14:58:13.364320 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:13.364321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:13.364331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:13.364374 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:13.364826 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.365314 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.366003 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:13.366257 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:13.366344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:13.368364 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:13.368365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:13.368365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:13.368372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:13.369733 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.370012 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.370013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.370013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.370060 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:13.370197 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:13.370197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:13.370197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:13.370319 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:13.370387 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:13.370478 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:58:13.370536 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 14:58:13.370852 5630 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59008
I20260928 14:58:13.371198 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5692
I20260928 14:58:13.376132 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5826
I20260928 14:58:13.380353 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5960
I20260928 14:58:13.384729 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 5598
2026-09-28T14:58:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1949 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[6102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[6102](info): set up 2 sockets
Sep 28 14:58:13 dist-test-slave-sf70 krb5kdc[6102](info): commencing operation
krb5kdc: starting...
W20260928 14:58:14.429395 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:14 dist-test-slave-sf70 krb5kdc[6102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:14.552668 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33649
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:33649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:14.646755 6115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:14.646968 6115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:14.647006 6115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:14.648214 6115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:14.648274 6115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:14.648300 6115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:14.648331 6115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:14.648355 6115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:14.649839 6115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:33649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:14.650106 6115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:14.650324 6115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:14.655088 6123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:14.655087 6120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:14.655122 6115 server_base.cc:1034] running on GCE node
W20260928 14:58:14.655081 6121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:14.655511 6115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:14.655789 6115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:14.656955 6115 hybrid_clock.cc:648] HybridClock initialized: now 1790607494656928 us; error 37 us; skew 500 ppm
Sep 28 14:58:14 dist-test-slave-sf70 krb5kdc[6102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:14.669013 6115 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:14.669414 6115 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:14.669490 6115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:14.669651 6115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:14.670744 6115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "c80199b5ab1e4ce0be76edad826b52c4"
format_stamp: "Formatted at 2026-09-28 14:58:14 on dist-test-slave-sf70"
server_key: "687f1153e18d909b89e757a1959c1f38"
server_key_iv: "31d0b2e68cbc0d69880de264d46828f0"
server_key_version: "encryptionkey@0"
I20260928 14:58:14.671126 6115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "c80199b5ab1e4ce0be76edad826b52c4"
format_stamp: "Formatted at 2026-09-28 14:58:14 on dist-test-slave-sf70"
server_key: "687f1153e18d909b89e757a1959c1f38"
server_key_iv: "31d0b2e68cbc0d69880de264d46828f0"
server_key_version: "encryptionkey@0"
I20260928 14:58:14.672586 6115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:14.673405 6129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:14.673607 6115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:14.673708 6115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "c80199b5ab1e4ce0be76edad826b52c4"
format_stamp: "Formatted at 2026-09-28 14:58:14 on dist-test-slave-sf70"
server_key: "687f1153e18d909b89e757a1959c1f38"
server_key_iv: "31d0b2e68cbc0d69880de264d46828f0"
server_key_version: "encryptionkey@0"
I20260928 14:58:14.673794 6115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:14.700917 6115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:14.703333 6115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:14.703527 6115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:14.707677 6115 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:33649
I20260928 14:58:14.707710 6181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:33649 every 8 connection(s)
I20260928 14:58:14.708065 6115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:14.709002 6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:14.711247 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4: Bootstrap starting.
I20260928 14:58:14.711798 6182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:14.712064 6182 log.cc:824] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:14.712677 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4: No bootstrap required, opened a new log
I20260928 14:58:14.714156 6182 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80199b5ab1e4ce0be76edad826b52c4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33649 } }
I20260928 14:58:14.714308 6182 raft_consensus.cc:374] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:14.714332 6182 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80199b5ab1e4ce0be76edad826b52c4, State: Initialized, Role: FOLLOWER
I20260928 14:58:14.714468 6182 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80199b5ab1e4ce0be76edad826b52c4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33649 } }
I20260928 14:58:14.714550 6182 raft_consensus.cc:388] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:14.714573 6182 raft_consensus.cc:482] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:14.714601 6182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:14.715376 6182 raft_consensus.cc:504] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80199b5ab1e4ce0be76edad826b52c4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33649 } }
I20260928 14:58:14.715461 6182 leader_election.cc:302] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c80199b5ab1e4ce0be76edad826b52c4; no voters:
I20260928 14:58:14.715584 6182 leader_election.cc:288] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:14.715669 6185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:14.715839 6182 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:14.715844 6185 raft_consensus.cc:686] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [term 1 LEADER]: Becoming Leader. State: Replica: c80199b5ab1e4ce0be76edad826b52c4, State: Running, Role: LEADER
I20260928 14:58:14.715984 6185 consensus_queue.cc:237] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80199b5ab1e4ce0be76edad826b52c4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33649 } }
I20260928 14:58:14.716361 6187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c80199b5ab1e4ce0be76edad826b52c4. Latest consensus state: current_term: 1 leader_uuid: "c80199b5ab1e4ce0be76edad826b52c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80199b5ab1e4ce0be76edad826b52c4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33649 } } }
I20260928 14:58:14.716396 6186 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c80199b5ab1e4ce0be76edad826b52c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80199b5ab1e4ce0be76edad826b52c4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33649 } } }
I20260928 14:58:14.716454 6187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:14.716490 6186 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:14.716684 6193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:14.717353 6193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:14.717631 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6115
I20260928 14:58:14.717729 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:14.717923 2035 external_mini_cluster.cc:1428] Setting key 42553b79cba7bab1a3cd7d8bbfb63512
I20260928 14:58:14.719123 6193 catalog_manager.cc:1347] Generated new cluster ID: 827cf0b627bb40438d772bd6ba5c905f
I20260928 14:58:14.719182 6193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:14 dist-test-slave-sf70 krb5kdc[6102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:14.729710 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:14.718424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.1.252.254:33649)
0928 14:58:14.718712 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:14.718722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:14.718760 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:58:14.718873 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:14.718878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:14.718922 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:14.719871 (+ 949us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:14.719991 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:14.721416 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:14.722541 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:14.723147 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:14.723240 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:14.726043 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:14.726047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:14.726047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:14.726069 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:14.728108 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:14.728492 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:14.728495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:14.728495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:14.728598 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:14.728797 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:14.728798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:14.728798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:14.729097 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:14.729269 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:14.729337 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:58:14.729486 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":65,"threads_started":1}
I20260928 14:58:14.735567 6193 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:14.736251 6193 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:14.739153 6193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4: Generated new TSK 0
I20260928 14:58:14.739321 6193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:14.768102 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33649
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:14.849136 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:14.849331 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:14.849355 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:14.850508 6208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:14.850548 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:14.850575 6208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:14.852064 6208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:14.852296 6208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:14.852538 6208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:14.857764 6216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:14.857762 6214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:14.857806 6213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:14.857977 6208 server_base.cc:1034] running on GCE node
I20260928 14:58:14.858213 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:14.858400 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:14.859548 6208 hybrid_clock.cc:648] HybridClock initialized: now 1790607494859539 us; error 35 us; skew 500 ppm
Sep 28 14:58:14 dist-test-slave-sf70 krb5kdc[6102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:14.862177 6208 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:14.862521 6208 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:14.862581 6208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:14.862673 6208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:14.863776 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "506b1701423c4f489407a00e0334ce60"
format_stamp: "Formatted at 2026-09-28 14:58:14 on dist-test-slave-sf70"
server_key: "4f51223c1d89fa3da4be2b8c09155130"
server_key_iv: "08672a18afdda17ca9ddc750964a055b"
server_key_version: "encryptionkey@0"
I20260928 14:58:14.864161 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "506b1701423c4f489407a00e0334ce60"
format_stamp: "Formatted at 2026-09-28 14:58:14 on dist-test-slave-sf70"
server_key: "4f51223c1d89fa3da4be2b8c09155130"
server_key_iv: "08672a18afdda17ca9ddc750964a055b"
server_key_version: "encryptionkey@0"
I20260928 14:58:14.865387 6208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:14.866058 6222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:14.866255 6208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:14.866330 6208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "506b1701423c4f489407a00e0334ce60"
format_stamp: "Formatted at 2026-09-28 14:58:14 on dist-test-slave-sf70"
server_key: "4f51223c1d89fa3da4be2b8c09155130"
server_key_iv: "08672a18afdda17ca9ddc750964a055b"
server_key_version: "encryptionkey@0"
I20260928 14:58:14.866422 6208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:14.912362 6208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:14.914808 6208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:14.914983 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:14.915242 6208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:14.915581 6208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:14.915637 6208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:14.915674 6208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:14.915704 6208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:14.921682 6208 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:44677
I20260928 14:58:14.921720 6335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:44677 every 8 connection(s)
I20260928 14:58:14.921993 6208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:14.923365 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6208
I20260928 14:58:14.923483 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:14.923669 2035 external_mini_cluster.cc:1428] Setting key 657b081637a3d0178e9401a6233f7b1a
Sep 28 14:58:14 dist-test-slave-sf70 krb5kdc[6102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:14.932468 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:14.922675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:46609 (local address 127.1.252.254:33649)
0928 14:58:14.922763 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:14.922766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:14.922967 (+ 201us) server_negotiation.cc:407] Connection header received
0928 14:58:14.923629 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:14.923631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:14.923641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:14.923693 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:14.924926 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:14.925416 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:14.926553 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:14.926870 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:14.926932 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:14.929335 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:14.929337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:14.929338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:14.929346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:14.931012 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:14.931448 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:14.931449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:14.931450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:14.931513 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:14.931759 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:14.931760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:14.931761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:14.931950 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:14.932026 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:14.932274 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 14:58:14.932360 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:58:14.932540 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:14.922653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33649 (local address 127.1.252.193:46609)
0928 14:58:14.922838 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:14.922874 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:14.923524 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:14.923772 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:14.923777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:14.924141 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:14.924817 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:14.924823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:14.925510 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:14.925511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:14.926457 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:14.926459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:14.926968 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:14.926969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:14.927121 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:14.927607 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:14.927617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:14.929242 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:14.931117 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:14.931123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:14.931125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:14.931354 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:14.931589 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:14.931590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:14.931590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:14.931688 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:14.932019 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:14.932024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:14.932134 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 14:58:14.932219 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:58:14.932342 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 14:58:14.933012 6336 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33649
I20260928 14:58:14.933123 6336 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:14.933280 6336 heartbeater.cc:511] Master 127.1.252.254:33649 requested a full tablet report, sending...
I20260928 14:58:14.933822 6146 ts_manager.cc:194] Registered new tserver with Master: 506b1701423c4f489407a00e0334ce60 (127.1.252.193:44677)
I20260928 14:58:14.934405 6146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:46609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:14.951332 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33649
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:15.032042 6341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:15.032232 6341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:15.032255 6341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:15.033383 6341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:15.033424 6341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:15.033452 6341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:15.034862 6341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:15.035048 6341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:15.035219 6341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:15.040187 6347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.040200 6346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.040187 6349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.040401 6341 server_base.cc:1034] running on GCE node
I20260928 14:58:15.040647 6341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:15.040892 6341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:15.042034 6341 hybrid_clock.cc:648] HybridClock initialized: now 1790607495042014 us; error 41 us; skew 500 ppm
Sep 28 14:58:15 dist-test-slave-sf70 krb5kdc[6102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:15.044516 6341 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:15.044885 6341 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:15.044945 6341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:15.045114 6341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:15.046186 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "2372ac6ae80149fc8551c41a2d1b54a0"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "073ec4bb7730c8ff5e6e9ad6335ebb10"
server_key_iv: "080398db28bba231d74bcc45eacb000a"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.046530 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "2372ac6ae80149fc8551c41a2d1b54a0"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "073ec4bb7730c8ff5e6e9ad6335ebb10"
server_key_iv: "080398db28bba231d74bcc45eacb000a"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.047793 6341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:58:15.048550 6355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.048725 6341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:15.048805 6341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "2372ac6ae80149fc8551c41a2d1b54a0"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "073ec4bb7730c8ff5e6e9ad6335ebb10"
server_key_iv: "080398db28bba231d74bcc45eacb000a"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.048857 6341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:15.058879 6341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:15.061041 6341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:15.061147 6341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:15.061316 6341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:15.061630 6341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:15.061663 6341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.061683 6341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:15.061697 6341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.067446 6341 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42299
I20260928 14:58:15.067520 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42299 every 8 connection(s)
I20260928 14:58:15.067765 6341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:58:15 dist-test-slave-sf70 krb5kdc[6102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:15.076046 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6341
I20260928 14:58:15.076171 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:15.076345 2035 external_mini_cluster.cc:1428] Setting key 2d14ee915d1ae2d57444b0fc1974913a
I20260928 14:58:15.076758 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.068454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:52361 (local address 127.1.252.254:33649)
0928 14:58:15.068496 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.068498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.068697 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:58:15.069244 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.069246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.069264 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.069308 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.070207 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.070764 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.071761 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.072055 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.072140 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.074009 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.074011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.074011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:15.074019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.075222 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:15.075619 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:15.075620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:15.075621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:15.075680 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:15.076050 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:15.076051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:15.076052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:15.076208 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.076288 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.076582 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.076669 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:15.076972 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.068421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33649 (local address 127.1.252.194:52361)
0928 14:58:15.068579 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.068612 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.069109 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.069385 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.069389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.069638 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.070092 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.070097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.070858 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.070859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.071655 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.071658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.072149 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.072150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.072280 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.072662 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:15.072669 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.073908 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.075307 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:15.075314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:15.075316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:15.075540 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:15.075760 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:15.075761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:15.075761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:15.075926 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:15.076278 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.076284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.076411 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.076560 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.076712 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 14:58:15.077414 6469 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33649
I20260928 14:58:15.077502 6469 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:15.077659 6469 heartbeater.cc:511] Master 127.1.252.254:33649 requested a full tablet report, sending...
I20260928 14:58:15.077996 6146 ts_manager.cc:194] Registered new tserver with Master: 2372ac6ae80149fc8551c41a2d1b54a0 (127.1.252.194:42299)
I20260928 14:58:15.078364 6146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:52361
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:15.100518 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33649
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:15.180640 6474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:15.180882 6474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:15.180924 6474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:15.182111 6474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:15.182178 6474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:15.182226 6474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:15.183756 6474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:15.184051 6474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:15.184291 6474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:58:15.189355 6474 server_base.cc:1034] running on GCE node
W20260928 14:58:15.189435 6480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.189352 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.189379 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.189718 6474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:15.189906 6474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:15.191043 6474 hybrid_clock.cc:648] HybridClock initialized: now 1790607495191032 us; error 30 us; skew 500 ppm
Sep 28 14:58:15 dist-test-slave-sf70 krb5kdc[6102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:15.193512 6474 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:15.193879 6474 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:15.193933 6474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:15.194101 6474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:15.195180 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "fe8d28d5bf6c423e8b20db16d41c0477"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "bb57bf7bdbe63af4ee236851d54f14cf"
server_key_iv: "6840dc3b4d4077110435e3182bae16db"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.195539 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "fe8d28d5bf6c423e8b20db16d41c0477"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "bb57bf7bdbe63af4ee236851d54f14cf"
server_key_iv: "6840dc3b4d4077110435e3182bae16db"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.196928 6474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:15.197701 6488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.197907 6474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:15.197973 6474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "fe8d28d5bf6c423e8b20db16d41c0477"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "bb57bf7bdbe63af4ee236851d54f14cf"
server_key_iv: "6840dc3b4d4077110435e3182bae16db"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.198125 6474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:15.214950 6474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:15.217254 6474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:15.217401 6474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:15.217635 6474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:15.217955 6474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:15.218011 6474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.218047 6474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:15.218075 6474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.223742 6474 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:44941
I20260928 14:58:15.223776 6601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:44941 every 8 connection(s)
I20260928 14:58:15.224098 6474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:15.224871 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6474
I20260928 14:58:15.225001 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:15.225194 2035 external_mini_cluster.cc:1428] Setting key 917d9551f1cc10dec409427bff653ee5
Sep 28 14:58:15 dist-test-slave-sf70 krb5kdc[6102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:15.232811 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.224813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:58831 (local address 127.1.252.254:33649)
0928 14:58:15.224858 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.224865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.225113 (+ 248us) server_negotiation.cc:407] Connection header received
0928 14:58:15.225847 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.225848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.225859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.225919 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.226874 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.227357 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.228379 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.228570 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.228621 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.230541 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.230542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.230542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:15.230549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.231692 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:15.232108 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:15.232109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:15.232109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:15.232161 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:15.232351 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:15.232351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:15.232351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:15.232450 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.232494 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.232695 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.232751 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:58:15.232999 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.224802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33649 (local address 127.1.252.195:58831)
0928 14:58:15.224990 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.225038 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.225740 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.225988 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.225992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.226332 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.226799 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.226804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.227444 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.227445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.228309 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.228312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.228640 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.228641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.228734 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.229128 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:15.229136 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.230461 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.231760 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:15.231766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:15.231767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:15.232045 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:15.232218 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:15.232219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:15.232219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:15.232311 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:15.232500 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.232504 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.232606 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.232683 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.232793 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260928 14:58:15.233428 6602 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33649
I20260928 14:58:15.233515 6602 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:15.233618 6602 heartbeater.cc:511] Master 127.1.252.254:33649 requested a full tablet report, sending...
I20260928 14:58:15.233980 6146 ts_manager.cc:194] Registered new tserver with Master: fe8d28d5bf6c423e8b20db16d41c0477 (127.1.252.195:44941)
I20260928 14:58:15.234441 6146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:58831
I20260928 14:58:15.236969 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:58:15.243980 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.238217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.1.252.254:33649)
0928 14:58:15.238283 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.238285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.238340 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:58:15.238411 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.238411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.238421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.238475 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.239019 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.239519 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.240310 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.240510 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.240592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.241634 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.241635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.241636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:15.241642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.242768 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:15.243106 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:15.243107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:15.243107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:15.243167 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:15.243363 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:15.243364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:15.243364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:15.243511 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.243613 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.243770 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.243846 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:58:15.245422 6146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:15.246968 6146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:15.253674 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.249189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:44941 (local address 127.0.0.1:45058)
0928 14:58:15.249497 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.249535 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.249631 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.249814 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.249817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.249820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:15.249991 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.249994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.251303 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.251304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.252672 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.252675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.253359 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.253360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.253449 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.253464 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.253531 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.253590 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":59,"thread_start_us":60,"threads_started":1}
I20260928 14:58:15.253744 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.249551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.1.252.193:44677)
0928 14:58:15.249731 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.249741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.249844 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:58:15.249920 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.249923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.249941 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.250019 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:15.250233 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.251346 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.252686 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.253420 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.253493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.253544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.253610 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.253652 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260928 14:58:15.253823 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.249211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42299 (local address 127.0.0.1:49318)
0928 14:58:15.249584 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.249595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.249649 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.250216 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.250217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.250219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:15.250251 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.250252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.251193 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.251194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.252677 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.252679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.253582 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.253583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.253646 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.253650 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.253700 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.253742 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":99,"thread_start_us":79,"threads_started":1}
I20260928 14:58:15.254016 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.249328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.1.252.195:44941)
0928 14:58:15.249529 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.249535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.249615 (+ 80us) server_negotiation.cc:407] Connection header received
0928 14:58:15.249716 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.249718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.249733 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.249807 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:15.250076 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.251199 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.252778 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.253259 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.253347 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.253413 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.253877 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.253942 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260928 14:58:15.254066 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.249318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.1.252.194:42299)
0928 14:58:15.249988 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.249993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.250001 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:15.250041 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.250043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.250056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.250124 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:15.250334 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.251093 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.252759 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.253420 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.253493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.253544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.253929 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.253980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":81,"threads_started":1}
I20260928 14:58:15.254348 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.249394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:44677 (local address 127.0.0.1:53776)
0928 14:58:15.249775 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.249782 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.249851 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.250110 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.250111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.250113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:15.250165 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.250167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.251459 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.251460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.252605 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.252607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.253489 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.253490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.253548 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.253551 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.253600 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.253631 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":71,"threads_started":1}
I20260928 14:58:15.254797 6536 tablet_service.cc:1467] Processing CreateTablet for tablet 78a77cc75aae4ea0a766c9340eb89070 (DEFAULT_TABLE table=test-table [id=992cb2eb24cb4c6a88d0e29bd7090a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:15.254807 6403 tablet_service.cc:1467] Processing CreateTablet for tablet 78a77cc75aae4ea0a766c9340eb89070 (DEFAULT_TABLE table=test-table [id=992cb2eb24cb4c6a88d0e29bd7090a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:15.254968 6270 tablet_service.cc:1467] Processing CreateTablet for tablet 78a77cc75aae4ea0a766c9340eb89070 (DEFAULT_TABLE table=test-table [id=992cb2eb24cb4c6a88d0e29bd7090a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:15.255153 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a77cc75aae4ea0a766c9340eb89070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.255156 6403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a77cc75aae4ea0a766c9340eb89070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.255246 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a77cc75aae4ea0a766c9340eb89070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.257833 6619 tablet_bootstrap.cc:492] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Bootstrap starting.
I20260928 14:58:15.258421 6620 tablet_bootstrap.cc:492] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Bootstrap starting.
I20260928 14:58:15.258544 6619 tablet_bootstrap.cc:654] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.258682 6621 tablet_bootstrap.cc:492] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60: Bootstrap starting.
I20260928 14:58:15.259116 6620 tablet_bootstrap.cc:654] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.259264 6621 tablet_bootstrap.cc:654] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.259265 6619 log.cc:824] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.259722 6620 log.cc:824] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.259958 6621 log.cc:824] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.260205 6619 tablet_bootstrap.cc:492] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: No bootstrap required, opened a new log
I20260928 14:58:15.260280 6619 ts_tablet_manager.cc:1397] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:58:15.260669 6621 tablet_bootstrap.cc:492] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60: No bootstrap required, opened a new log
I20260928 14:58:15.260741 6621 ts_tablet_manager.cc:1397] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:15.260504 6620 tablet_bootstrap.cc:492] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: No bootstrap required, opened a new log
I20260928 14:58:15.260885 6620 ts_tablet_manager.cc:1397] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:58:15.261777 6619 raft_consensus.cc:348] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.261930 6619 raft_consensus.cc:374] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.261974 6619 raft_consensus.cc:729] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8d28d5bf6c423e8b20db16d41c0477, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.262104 6619 consensus_queue.cc:260] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.262445 6602 heartbeater.cc:503] Master 127.1.252.254:33649 was elected leader, sending a full tablet report...
I20260928 14:58:15.262504 6619 ts_tablet_manager.cc:1428] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:58:15.262749 6621 raft_consensus.cc:348] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.262933 6621 raft_consensus.cc:374] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.262979 6621 raft_consensus.cc:729] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b1701423c4f489407a00e0334ce60, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.262840 6620 raft_consensus.cc:348] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.263018 6620 raft_consensus.cc:374] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.263095 6620 raft_consensus.cc:729] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2372ac6ae80149fc8551c41a2d1b54a0, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.263083 6621 consensus_queue.cc:260] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.263252 6620 consensus_queue.cc:260] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.263350 6621 ts_tablet_manager.cc:1428] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:58:15.263466 6336 heartbeater.cc:503] Master 127.1.252.254:33649 was elected leader, sending a full tablet report...
I20260928 14:58:15.263567 6469 heartbeater.cc:503] Master 127.1.252.254:33649 was elected leader, sending a full tablet report...
I20260928 14:58:15.263818 6620 ts_tablet_manager.cc:1428] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 14:58:15.318320 6470 tablet.cc:2369] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:15.362620 6626 raft_consensus.cc:482] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:15.362743 6626 raft_consensus.cc:504] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.363081 6626 leader_election.cc:288] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8d28d5bf6c423e8b20db16d41c0477 (127.1.252.195:44941), 2372ac6ae80149fc8551c41a2d1b54a0 (127.1.252.194:42299)
I20260928 14:58:15.367112 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.363365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:35169 (local address 127.1.252.194:42299)
0928 14:58:15.363426 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.363428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.363568 (+ 140us) server_negotiation.cc:407] Connection header received
0928 14:58:15.363676 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.363677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.363688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.363744 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:15.363960 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.364642 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.366121 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.366705 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.366785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.366794 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.367000 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.367042 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:58:15.367112 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.363312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:36189 (local address 127.1.252.195:44941)
0928 14:58:15.363472 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.363474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.363485 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:58:15.363516 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.363517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.363528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.363587 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:15.363761 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.364594 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.366121 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.366706 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.366785 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.366794 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.367000 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.367042 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260928 14:58:15.367117 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.363258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:44941 (local address 127.1.252.193:36189)
0928 14:58:15.363316 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.363341 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.363387 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.363600 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.363602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.363604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:15.363678 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.363680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.364699 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.364700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.366029 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.366031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.366796 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.366797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.366892 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.366897 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.366962 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.367016 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":68}
I20260928 14:58:15.367123 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.363276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42299 (local address 127.1.252.193:35169)
0928 14:58:15.363487 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.363497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.363569 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.363771 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.363772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.363774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:15.363834 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.363836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.364719 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.364719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.366029 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:15.366031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.366796 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.366797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:15.366892 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.366897 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.366969 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.367015 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":33,"thread_start_us":119,"threads_started":1}
I20260928 14:58:15.367643 6556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a77cc75aae4ea0a766c9340eb89070" candidate_uuid: "506b1701423c4f489407a00e0334ce60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" is_pre_election: true
I20260928 14:58:15.367806 6556 raft_consensus.cc:2454] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506b1701423c4f489407a00e0334ce60 in term 0.
I20260928 14:58:15.368041 6224 leader_election.cc:302] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b1701423c4f489407a00e0334ce60, fe8d28d5bf6c423e8b20db16d41c0477; no voters:
I20260928 14:58:15.368111 6423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a77cc75aae4ea0a766c9340eb89070" candidate_uuid: "506b1701423c4f489407a00e0334ce60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" is_pre_election: true
I20260928 14:58:15.368199 6626 raft_consensus.cc:2781] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:15.368252 6626 raft_consensus.cc:482] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:15.368261 6423 raft_consensus.cc:2454] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506b1701423c4f489407a00e0334ce60 in term 0.
I20260928 14:58:15.368283 6626 raft_consensus.cc:3037] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:15.369102 6626 raft_consensus.cc:504] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.369259 6626 leader_election.cc:288] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [CANDIDATE]: Term 1 election: Requested vote from peers fe8d28d5bf6c423e8b20db16d41c0477 (127.1.252.195:44941), 2372ac6ae80149fc8551c41a2d1b54a0 (127.1.252.194:42299)
I20260928 14:58:15.369417 6423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a77cc75aae4ea0a766c9340eb89070" candidate_uuid: "506b1701423c4f489407a00e0334ce60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2372ac6ae80149fc8551c41a2d1b54a0"
I20260928 14:58:15.369458 6556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a77cc75aae4ea0a766c9340eb89070" candidate_uuid: "506b1701423c4f489407a00e0334ce60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8d28d5bf6c423e8b20db16d41c0477"
I20260928 14:58:15.369505 6423 raft_consensus.cc:3037] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:15.369544 6556 raft_consensus.cc:3037] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:15.370306 6556 raft_consensus.cc:2454] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506b1701423c4f489407a00e0334ce60 in term 1.
I20260928 14:58:15.370322 6423 raft_consensus.cc:2454] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506b1701423c4f489407a00e0334ce60 in term 1.
I20260928 14:58:15.370499 6226 leader_election.cc:302] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2372ac6ae80149fc8551c41a2d1b54a0, 506b1701423c4f489407a00e0334ce60; no voters:
I20260928 14:58:15.370605 6626 raft_consensus.cc:2781] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:15.370711 6626 raft_consensus.cc:686] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [term 1 LEADER]: Becoming Leader. State: Replica: 506b1701423c4f489407a00e0334ce60, State: Running, Role: LEADER
I20260928 14:58:15.370800 6626 consensus_queue.cc:237] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } }
I20260928 14:58:15.371656 6146 catalog_manager.cc:5495] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 506b1701423c4f489407a00e0334ce60 (127.1.252.193). New cstate: current_term: 1 leader_uuid: "506b1701423c4f489407a00e0334ce60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506b1701423c4f489407a00e0334ce60" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 44677 } health_report { overall_health: HEALTHY } } }
I20260928 14:58:15.397044 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.394638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.1.252.193:44677)
0928 14:58:15.394710 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.394712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.394721 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:15.394771 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.394772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.394789 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.394867 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:15.394991 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.395459 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.396245 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:15.396439 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:15.396525 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:15.396689 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:15.396793 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:15.396889 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.396914 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.396933 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:58:15.398891 6556 raft_consensus.cc:1261] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 1 FOLLOWER]: Refusing update from remote peer 506b1701423c4f489407a00e0334ce60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:15.398952 6423 raft_consensus.cc:1261] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 1 FOLLOWER]: Refusing update from remote peer 506b1701423c4f489407a00e0334ce60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:15.399303 6629 consensus_queue.cc:1035] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8d28d5bf6c423e8b20db16d41c0477" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:15.399431 6626 consensus_queue.cc:1035] T 78a77cc75aae4ea0a766c9340eb89070 P 506b1701423c4f489407a00e0334ce60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2372ac6ae80149fc8551c41a2d1b54a0" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:15.402395 6634 mvcc.cc:205] Tried to move back new op lower bound from 7334328301151170560 to 7334328301039587328. Current Snapshot: MvccSnapshot[applied={T|T < 7334328301151170560}]
I20260928 14:58:15.404752 6145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:15.404843 6145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:15.405644 6145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c80199b5ab1e4ce0be76edad826b52c4: Sending DeleteTablet for 3 replicas of tablet 78a77cc75aae4ea0a766c9340eb89070
I20260928 14:58:15.405951 6536 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a77cc75aae4ea0a766c9340eb89070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:15 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:45058
I20260928 14:58:15.406035 6403 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a77cc75aae4ea0a766c9340eb89070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:15 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:49318
I20260928 14:58:15.406234 6270 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a77cc75aae4ea0a766c9340eb89070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:15 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:53776
I20260928 14:58:15.406250 6643 tablet_replica.cc:330] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: stopping tablet replica
I20260928 14:58:15.406272 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6208
I20260928 14:58:15.406347 6643 raft_consensus.cc:2229] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:15.406438 6643 raft_consensus.cc:2258] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:15.407276 6643 ts_tablet_manager.cc:1905] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:15.408661 6642 tablet_replica.cc:330] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: stopping tablet replica
I20260928 14:58:15.408780 6642 raft_consensus.cc:2229] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:15.408875 6642 raft_consensus.cc:2258] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:15.409231 6642 ts_tablet_manager.cc:1905] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:15.409341 6643 ts_tablet_manager.cc:1918] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:15.409420 6643 log.cc:1196] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/78a77cc75aae4ea0a766c9340eb89070
I20260928 14:58:15.409689 6643 ts_tablet_manager.cc:1939] T 78a77cc75aae4ea0a766c9340eb89070 P 2372ac6ae80149fc8551c41a2d1b54a0: Deleting consensus metadata
I20260928 14:58:15.410173 6133 catalog_manager.cc:4841] TS 2372ac6ae80149fc8551c41a2d1b54a0 (127.1.252.194:42299): tablet 78a77cc75aae4ea0a766c9340eb89070 (table test-table [id=992cb2eb24cb4c6a88d0e29bd7090a3b]) successfully deleted
I20260928 14:58:15.410660 6642 ts_tablet_manager.cc:1918] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:15.410748 6642 log.cc:1196] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/78a77cc75aae4ea0a766c9340eb89070
I20260928 14:58:15.410990 6642 ts_tablet_manager.cc:1939] T 78a77cc75aae4ea0a766c9340eb89070 P fe8d28d5bf6c423e8b20db16d41c0477: Deleting consensus metadata
I20260928 14:58:15.411598 6131 catalog_manager.cc:4841] TS fe8d28d5bf6c423e8b20db16d41c0477 (127.1.252.195:44941): tablet 78a77cc75aae4ea0a766c9340eb89070 (table test-table [id=992cb2eb24cb4c6a88d0e29bd7090a3b]) successfully deleted
W20260928 14:58:15.412794 6133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:44677 (error 108)
I20260928 14:58:15.412866 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6341
I20260928 14:58:15.413321 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.413212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:44677 (local address 127.0.0.1:53806)
0928 14:58:15.413253 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.413281 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 14:58:15.413442 6133 catalog_manager.cc:4584] TS 506b1701423c4f489407a00e0334ce60 (127.1.252.193:44677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78a77cc75aae4ea0a766c9340eb89070: Network error: Client connection negotiation failed: client connection to 127.1.252.193:44677: connect: Connection refused (error 111)
I20260928 14:58:15.418413 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6474
I20260928 14:58:15.423282 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6115
2026-09-28T14:58:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2039 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:15Z Disabled control of system clock
I20260928 14:58:15.450428 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46007
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:46007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:58:15.533272 6648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:15.533475 6648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:15.533496 6648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:15.534664 6648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:15.534708 6648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:15.534720 6648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:15.534734 6648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:15.534785 6648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:15.536125 6648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:46007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:15.536367 6648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:15.536655 6648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:15.541247 6656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.541257 6654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.541440 6648 server_base.cc:1034] running on GCE node
W20260928 14:58:15.541455 6653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.541747 6648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:15.542024 6648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:15.543156 6648 hybrid_clock.cc:648] HybridClock initialized: now 1790607495543146 us; error 27 us; skew 500 ppm
I20260928 14:58:15.544516 6648 webserver.cc:460] Webserver started at http://127.1.252.254:34973/ using document root <none> and password file <none>
I20260928 14:58:15.544750 6648 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:15.544797 6648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:15.544951 6648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:15.545984 6648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "2ac72969fb684bab9b1247ee226aadef"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "b7e73e9b63a04b637b4077b5fb7030b9"
server_key_iv: "6c7937a92739a899d6659c75798211ed"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.546342 6648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "2ac72969fb684bab9b1247ee226aadef"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "b7e73e9b63a04b637b4077b5fb7030b9"
server_key_iv: "6c7937a92739a899d6659c75798211ed"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.547578 6648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:15.548326 6662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.548491 6648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.548578 6648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "2ac72969fb684bab9b1247ee226aadef"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "b7e73e9b63a04b637b4077b5fb7030b9"
server_key_iv: "6c7937a92739a899d6659c75798211ed"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.548651 6648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:15.551816 6648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:15.552089 6648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:15.552224 6648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:15.555794 6648 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:46007
I20260928 14:58:15.555823 6714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:46007 every 8 connection(s)
I20260928 14:58:15.556161 6648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:15.557067 6715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.559175 6715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef: Bootstrap starting.
I20260928 14:58:15.559729 6715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.560065 6715 log.cc:824] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.560765 6715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef: No bootstrap required, opened a new log
I20260928 14:58:15.562371 6715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac72969fb684bab9b1247ee226aadef" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46007 } }
I20260928 14:58:15.562522 6715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.562546 6715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac72969fb684bab9b1247ee226aadef, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.562700 6715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac72969fb684bab9b1247ee226aadef" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46007 } }
I20260928 14:58:15.562768 6715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:15.562789 6715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:15.562848 6715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:15.563360 6715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac72969fb684bab9b1247ee226aadef" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46007 } }
I20260928 14:58:15.563478 6715 leader_election.cc:302] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ac72969fb684bab9b1247ee226aadef; no voters:
I20260928 14:58:15.563647 6715 leader_election.cc:288] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:15.563704 6718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:15.563862 6718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [term 1 LEADER]: Becoming Leader. State: Replica: 2ac72969fb684bab9b1247ee226aadef, State: Running, Role: LEADER
I20260928 14:58:15.563925 6715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:15.563995 6718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac72969fb684bab9b1247ee226aadef" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46007 } }
I20260928 14:58:15.564352 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6648
I20260928 14:58:15.564561 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:15.564440 6720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ac72969fb684bab9b1247ee226aadef. Latest consensus state: current_term: 1 leader_uuid: "2ac72969fb684bab9b1247ee226aadef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac72969fb684bab9b1247ee226aadef" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46007 } } }
I20260928 14:58:15.564603 6720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:15.564739 2035 external_mini_cluster.cc:1428] Setting key 9dcd14b1498a6149516a5d9fd15a1a93
I20260928 14:58:15.565091 6727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:15.565608 6727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:15.565636 6719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ac72969fb684bab9b1247ee226aadef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac72969fb684bab9b1247ee226aadef" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46007 } } }
I20260928 14:58:15.565984 6719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:15.567255 6727 catalog_manager.cc:1347] Generated new cluster ID: b5acc611d3384793903f3623226908ba
I20260928 14:58:15.567307 6727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:15.567548 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.565343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.1.252.254:46007)
0928 14:58:15.565772 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.565780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.565814 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:58:15.565921 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.565928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.565970 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.566106 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.567024 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.567028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.567028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.567034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.567083 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.567098 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.567155 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.567330 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.567387 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":93,"threads_started":1}
I20260928 14:58:15.581135 6727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:15.581308 6727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:15.589553 6727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef: Generated new TSK 0
I20260928 14:58:15.589753 6727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:15.592378 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46007
--builtin_ntp_servers=127.1.252.212:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:58:15.678740 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:15.678925 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:15.678947 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:15.680164 6739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:15.680203 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:15.680229 6739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:15.681552 6739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:15.681777 6739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:15.682086 6739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:15.687716 6747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.687732 6745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.687790 6744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.688215 6739 server_base.cc:1034] running on GCE node
I20260928 14:58:15.688434 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:15.688678 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:15.689839 6739 hybrid_clock.cc:648] HybridClock initialized: now 1790607495689825 us; error 30 us; skew 500 ppm
I20260928 14:58:15.691174 6739 webserver.cc:460] Webserver started at http://127.1.252.193:45853/ using document root <none> and password file <none>
I20260928 14:58:15.691484 6739 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:15.691557 6739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:15.691715 6739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:15.692809 6739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "2c5523866ee944a2b90ae085b93e5ec9"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "6982f90ce4f4d6ad96206a17b18c8c96"
server_key_iv: "7a79d6373d63fcfe9292fb7248c91a31"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.693176 6739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "2c5523866ee944a2b90ae085b93e5ec9"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "6982f90ce4f4d6ad96206a17b18c8c96"
server_key_iv: "7a79d6373d63fcfe9292fb7248c91a31"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.694525 6739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:15.695307 6753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.695551 6739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:15.695641 6739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "2c5523866ee944a2b90ae085b93e5ec9"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "6982f90ce4f4d6ad96206a17b18c8c96"
server_key_iv: "7a79d6373d63fcfe9292fb7248c91a31"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.695725 6739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:15.698980 6739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:15.699258 6739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:15.699409 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:15.699643 6739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:15.700006 6739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:15.700059 6739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.700099 6739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:15.700126 6739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.705988 6739 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:38751
I20260928 14:58:15.706028 6866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:38751 every 8 connection(s)
I20260928 14:58:15.706342 6739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:15.707496 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6739
I20260928 14:58:15.707600 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:15.707778 2035 external_mini_cluster.cc:1428] Setting key 43a8d326cedefc87bc0a403d9ba6a6bc
I20260928 14:58:15.708757 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46007
--builtin_ntp_servers=127.1.252.212:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:58:15.709544 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.707070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:35057 (local address 127.1.252.254:46007)
0928 14:58:15.707134 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.707136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.707358 (+ 222us) server_negotiation.cc:407] Connection header received
0928 14:58:15.707862 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.707864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.707871 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.707977 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.709273 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.709274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.709275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.709280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.709298 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.709304 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.709371 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.709431 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.709445 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:58:15.709641 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.707056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:46007 (local address 127.1.252.193:35057)
0928 14:58:15.707233 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.707263 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.707753 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.708053 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.708057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.708067 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.709130 (+ 1063us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:15.709131 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.709158 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:15.709186 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.709369 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.709371 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.709373 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.709422 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.709466 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260928 14:58:15.710183 6867 heartbeater.cc:347] Connected to a master server at 127.1.252.254:46007
I20260928 14:58:15.710273 6867 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:15.710440 6867 heartbeater.cc:511] Master 127.1.252.254:46007 requested a full tablet report, sending...
I20260928 14:58:15.710963 6679 ts_manager.cc:194] Registered new tserver with Master: 2c5523866ee944a2b90ae085b93e5ec9 (127.1.252.193:38751)
W20260928 14:58:15.788215 6870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:15.788406 6870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:15.788429 6870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:15.789492 6870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:15.789523 6870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:15.789548 6870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:15.790778 6870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:15.790949 6870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:15.791193 6870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:15.796139 6875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.796154 6876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.796283 6878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.796370 6870 server_base.cc:1034] running on GCE node
I20260928 14:58:15.796569 6870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:15.796808 6870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:15.797974 6870 hybrid_clock.cc:648] HybridClock initialized: now 1790607495797969 us; error 35 us; skew 500 ppm
I20260928 14:58:15.799271 6870 webserver.cc:460] Webserver started at http://127.1.252.194:35205/ using document root <none> and password file <none>
I20260928 14:58:15.799523 6870 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:15.799568 6870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:15.799701 6870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:15.800966 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "467768b6a41440d68ccf709788d81956"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "133cbb9edc1b6390a580de57641bb0fc"
server_key_iv: "fcea8fc1be20b837c00a36bd9d640895"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.801338 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "467768b6a41440d68ccf709788d81956"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "133cbb9edc1b6390a580de57641bb0fc"
server_key_iv: "fcea8fc1be20b837c00a36bd9d640895"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.802577 6870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:15.803258 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.803424 6870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:15.803522 6870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "467768b6a41440d68ccf709788d81956"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "133cbb9edc1b6390a580de57641bb0fc"
server_key_iv: "fcea8fc1be20b837c00a36bd9d640895"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.803609 6870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:15.806782 6870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:15.807049 6870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:15.807174 6870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:15.807420 6870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:15.807722 6870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:15.807778 6870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.807816 6870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:15.807844 6870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.813390 6870 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:33841
I20260928 14:58:15.813455 6997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:33841 every 8 connection(s)
I20260928 14:58:15.813782 6870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:15.816660 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.814491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:60481 (local address 127.1.252.254:46007)
0928 14:58:15.814563 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.814566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.814797 (+ 231us) server_negotiation.cc:407] Connection header received
0928 14:58:15.815216 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.815217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.815223 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.815275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.816473 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.816474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.816474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.816478 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.816493 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.816498 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.816547 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.816597 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.816609 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:58:15.816864 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.814473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:46007 (local address 127.1.252.194:60481)
0928 14:58:15.814687 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.814718 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.815138 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.815336 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.815339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.815347 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.816336 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:15.816337 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.816364 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:15.816392 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.816544 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.816546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.816548 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.816596 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.816640 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260928 14:58:15.817373 6998 heartbeater.cc:347] Connected to a master server at 127.1.252.254:46007
I20260928 14:58:15.817473 6998 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:15.817615 6998 heartbeater.cc:511] Master 127.1.252.254:46007 requested a full tablet report, sending...
I20260928 14:58:15.818018 6679 ts_manager.cc:194] Registered new tserver with Master: 467768b6a41440d68ccf709788d81956 (127.1.252.194:33841)
I20260928 14:58:15.823808 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 6870
I20260928 14:58:15.823949 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:15.824124 2035 external_mini_cluster.cc:1428] Setting key 391691b4f63149ba8faaf47d4e319ad6
I20260928 14:58:15.825064 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46007
--builtin_ntp_servers=127.1.252.212:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:58:15.909332 7001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:15.909534 7001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:15.909557 7001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:15.910708 7001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:15.910749 7001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:15.910802 7001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:15.912196 7001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:15.912469 7001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:15.912757 7001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:15.917596 7009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.917575 7006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:15.917579 7007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:15.917843 7001 server_base.cc:1034] running on GCE node
I20260928 14:58:15.918099 7001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:15.918329 7001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:15.919471 7001 hybrid_clock.cc:648] HybridClock initialized: now 1790607495919458 us; error 32 us; skew 500 ppm
I20260928 14:58:15.920665 7001 webserver.cc:460] Webserver started at http://127.1.252.195:40151/ using document root <none> and password file <none>
I20260928 14:58:15.920909 7001 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:15.920955 7001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:15.921118 7001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:15.922128 7001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "414ed6499fdd4ddbb5028c2aa6034c73"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "aa15e55fac11fdb347192989cfae398f"
server_key_iv: "a15165609e986b7b4a78728b2aa16b53"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.922490 7001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "414ed6499fdd4ddbb5028c2aa6034c73"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "aa15e55fac11fdb347192989cfae398f"
server_key_iv: "a15165609e986b7b4a78728b2aa16b53"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.923772 7001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:15.924723 7015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.924995 7001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:15.925082 7001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "414ed6499fdd4ddbb5028c2aa6034c73"
format_stamp: "Formatted at 2026-09-28 14:58:15 on dist-test-slave-sf70"
server_key: "aa15e55fac11fdb347192989cfae398f"
server_key_iv: "a15165609e986b7b4a78728b2aa16b53"
server_key_version: "encryptionkey@0"
I20260928 14:58:15.925156 7001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:15.928406 7001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:15.928664 7001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:15.928794 7001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:15.929011 7001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:15.929301 7001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:15.929334 7001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.929354 7001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:15.929397 7001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:15.935102 7001 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:41041
I20260928 14:58:15.935132 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:41041 every 8 connection(s)
I20260928 14:58:15.935433 7001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:15.938207 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.936133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:58607 (local address 127.1.252.254:46007)
0928 14:58:15.936173 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.936175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.936393 (+ 218us) server_negotiation.cc:407] Connection header received
0928 14:58:15.936833 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.936834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.936839 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.936881 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.937992 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.937992 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.937993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.937995 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.938006 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.938010 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.938064 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.938142 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.938155 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:15.938452 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.936092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:46007 (local address 127.1.252.195:58607)
0928 14:58:15.936270 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.936308 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.936737 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.936941 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.936944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.936951 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.937871 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:15.937872 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.937899 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:15.937924 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.938049 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.938051 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.938053 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.938153 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.938212 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 14:58:15.938922 7130 heartbeater.cc:347] Connected to a master server at 127.1.252.254:46007
I20260928 14:58:15.939018 7130 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:15.939152 7130 heartbeater.cc:511] Master 127.1.252.254:46007 requested a full tablet report, sending...
I20260928 14:58:15.939417 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7001
I20260928 14:58:15.939535 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:15.939530 6679 ts_manager.cc:194] Registered new tserver with Master: 414ed6499fdd4ddbb5028c2aa6034c73 (127.1.252.195:41041)
I20260928 14:58:15.939704 2035 external_mini_cluster.cc:1428] Setting key 803fcf75863bd7996d3303a3e58413a5
I20260928 14:58:15.940573 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:58:15.942854 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.941863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.1.252.254:46007)
0928 14:58:15.941916 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.941918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.942045 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:58:15.942082 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.942082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.942086 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.942128 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.942669 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.942669 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.942670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.942672 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.942683 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.942686 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.942720 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.942752 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.942763 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:58:15.944514 6679 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:15.946099 6679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:15.949851 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.948342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:33841 (local address 127.0.0.1:51602)
0928 14:58:15.948583 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.948619 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.948672 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.948878 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.948884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.948890 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.949358 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:15.949359 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.949374 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:15.949395 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.949662 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.949669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.949672 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.949754 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.949768 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260928 14:58:15.949877 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.948561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.1.252.194:33841)
0928 14:58:15.948720 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.948724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.948736 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:58:15.948764 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.948766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.948776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.948863 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.949490 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.949497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.949497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.949502 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.949557 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.949588 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.949651 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.949773 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.949791 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 14:58:15.950425 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.948344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:38751 (local address 127.0.0.1:47006)
0928 14:58:15.948583 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.948619 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.948672 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.949280 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.949281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.949300 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.949713 (+ 413us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:15.949714 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.949719 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:15.949748 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.950256 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.950257 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.950258 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.950325 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.950338 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":53,"threads_started":1}
I20260928 14:58:15.950647 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.948750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.1.252.193:38751)
0928 14:58:15.949144 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.949148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.949157 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:15.949183 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.949185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.949194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.949261 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.949835 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.949842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.949842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.949847 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.949879 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.950188 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.950261 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.950528 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.950546 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":87,"threads_started":1}
I20260928 14:58:15.951225 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.948392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.1.252.195:41041)
0928 14:58:15.948787 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:15.948791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:15.948930 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:58:15.949325 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:15.949330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:15.949342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:15.949414 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:15.950881 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:15.950887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:15.950887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:15.950891 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:15.950923 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:15.950947 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:15.951012 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:15.951131 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:58:15.951147 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":77,"threads_started":1}
I20260928 14:58:15.951215 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:15.948346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:41041 (local address 127.0.0.1:33136)
0928 14:58:15.948859 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:15.948869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:15.948970 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:15.950220 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:15.950221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:15.950223 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:15.950649 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:15.950649 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:15.950692 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:15.950801 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:15.951009 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:15.951010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:15.951011 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:58:15.951119 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:58:15.951159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":97,"threads_started":1}
I20260928 14:58:15.951836 7063 tablet_service.cc:1467] Processing CreateTablet for tablet b4f6d6f524804404bfd9368a8581b4dd (DEFAULT_TABLE table=test-table [id=8b4dbc275c8a451794d5d77fac53ba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:15.952196 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f6d6f524804404bfd9368a8581b4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.952217 6932 tablet_service.cc:1467] Processing CreateTablet for tablet b4f6d6f524804404bfd9368a8581b4dd (DEFAULT_TABLE table=test-table [id=8b4dbc275c8a451794d5d77fac53ba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:15.952543 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f6d6f524804404bfd9368a8581b4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.955091 7144 tablet_bootstrap.cc:492] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Bootstrap starting.
I20260928 14:58:15.955448 7145 tablet_bootstrap.cc:492] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Bootstrap starting.
I20260928 14:58:15.955806 7144 tablet_bootstrap.cc:654] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.955739 6801 tablet_service.cc:1467] Processing CreateTablet for tablet b4f6d6f524804404bfd9368a8581b4dd (DEFAULT_TABLE table=test-table [id=8b4dbc275c8a451794d5d77fac53ba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:15.956065 6801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f6d6f524804404bfd9368a8581b4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:15.956111 7145 tablet_bootstrap.cc:654] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.956240 7144 log.cc:824] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.956409 7145 log.cc:824] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.957026 7144 tablet_bootstrap.cc:492] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: No bootstrap required, opened a new log
I20260928 14:58:15.957054 7145 tablet_bootstrap.cc:492] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: No bootstrap required, opened a new log
I20260928 14:58:15.957093 7144 ts_tablet_manager.cc:1397] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:58:15.957129 7145 ts_tablet_manager.cc:1397] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:15.958830 7145 raft_consensus.cc:348] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:15.958976 7145 raft_consensus.cc:374] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.959017 7145 raft_consensus.cc:729] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467768b6a41440d68ccf709788d81956, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.959149 7145 consensus_queue.cc:260] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:15.959400 7145 ts_tablet_manager.cc:1428] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:15.959408 6998 heartbeater.cc:503] Master 127.1.252.254:46007 was elected leader, sending a full tablet report...
I20260928 14:58:15.959682 7149 tablet_bootstrap.cc:492] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: Bootstrap starting.
I20260928 14:58:15.959937 7144 raft_consensus.cc:348] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:15.960114 7144 raft_consensus.cc:374] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.960153 7144 raft_consensus.cc:729] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414ed6499fdd4ddbb5028c2aa6034c73, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.960279 7144 consensus_queue.cc:260] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:15.960352 7149 tablet_bootstrap.cc:654] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:15.960556 7144 ts_tablet_manager.cc:1428] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 14:58:15.960625 7130 heartbeater.cc:503] Master 127.1.252.254:46007 was elected leader, sending a full tablet report...
I20260928 14:58:15.960671 7149 log.cc:824] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:15.961411 7149 tablet_bootstrap.cc:492] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: No bootstrap required, opened a new log
I20260928 14:58:15.961490 7149 ts_tablet_manager.cc:1397] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:15.963196 7149 raft_consensus.cc:348] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:15.963333 7149 raft_consensus.cc:374] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:15.963377 7149 raft_consensus.cc:729] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5523866ee944a2b90ae085b93e5ec9, State: Initialized, Role: FOLLOWER
I20260928 14:58:15.963488 7149 consensus_queue.cc:260] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:15.963737 7149 ts_tablet_manager.cc:1428] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:58:15.963850 6867 heartbeater.cc:503] Master 127.1.252.254:46007 was elected leader, sending a full tablet report...
I20260928 14:58:16.051010 7150 raft_consensus.cc:482] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:16.051139 7150 raft_consensus.cc:504] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:16.051448 7150 leader_election.cc:288] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 467768b6a41440d68ccf709788d81956 (127.1.252.194:33841), 2c5523866ee944a2b90ae085b93e5ec9 (127.1.252.193:38751)
I20260928 14:58:16.052834 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.051730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:57831 (local address 127.1.252.194:33841)
0928 14:58:16.051816 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.051819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.051829 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:16.051858 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.051859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.051863 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.051978 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.052575 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.052576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.052577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.052582 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.052599 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.052603 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.052681 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.052748 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.052762 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:58:16.052831 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.051652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:33841 (local address 127.1.252.195:57831)
0928 14:58:16.051710 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.051723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.051763 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.052000 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:16.052001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:16.052003 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:16.052454 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:16.052454 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:16.052462 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:16.052473 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:16.052696 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:16.052697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:16.052698 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:58:16.052747 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:16.052762 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:58:16.052922 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.051730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:38917 (local address 127.1.252.193:38751)
0928 14:58:16.051776 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.051778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.051896 (+ 118us) server_negotiation.cc:407] Connection header received
0928 14:58:16.052027 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.052029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.052035 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.052090 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.052723 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.052724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.052724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.052728 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.052744 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.052748 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.052795 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.052841 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.052855 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:16.052946 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.051652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:38751 (local address 127.1.252.195:38917)
0928 14:58:16.051828 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.051837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.051963 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.052105 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:16.052106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:16.052108 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:16.052609 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:16.052609 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:16.052616 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:16.052628 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:16.052791 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:16.052792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:16.052792 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 14:58:16.052841 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:16.052855 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":13,"thread_start_us":106,"threads_started":1}
I20260928 14:58:16.053369 6821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f6d6f524804404bfd9368a8581b4dd" candidate_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5523866ee944a2b90ae085b93e5ec9" is_pre_election: true
I20260928 14:58:16.053537 6821 raft_consensus.cc:2454] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 414ed6499fdd4ddbb5028c2aa6034c73 in term 0.
I20260928 14:58:16.053521 6952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f6d6f524804404bfd9368a8581b4dd" candidate_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467768b6a41440d68ccf709788d81956" is_pre_election: true
I20260928 14:58:16.053658 6952 raft_consensus.cc:2454] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 414ed6499fdd4ddbb5028c2aa6034c73 in term 0.
I20260928 14:58:16.053803 7017 leader_election.cc:302] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c5523866ee944a2b90ae085b93e5ec9, 414ed6499fdd4ddbb5028c2aa6034c73; no voters:
I20260928 14:58:16.053923 7150 raft_consensus.cc:2781] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:16.053993 7150 raft_consensus.cc:482] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:16.054025 7150 raft_consensus.cc:3037] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.054759 7150 raft_consensus.cc:504] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:16.054898 7150 leader_election.cc:288] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [CANDIDATE]: Term 1 election: Requested vote from peers 467768b6a41440d68ccf709788d81956 (127.1.252.194:33841), 2c5523866ee944a2b90ae085b93e5ec9 (127.1.252.193:38751)
I20260928 14:58:16.055045 6821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f6d6f524804404bfd9368a8581b4dd" candidate_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5523866ee944a2b90ae085b93e5ec9"
I20260928 14:58:16.055047 6952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f6d6f524804404bfd9368a8581b4dd" candidate_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467768b6a41440d68ccf709788d81956"
I20260928 14:58:16.055130 6821 raft_consensus.cc:3037] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.055135 6952 raft_consensus.cc:3037] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.056054 6821 raft_consensus.cc:2454] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 414ed6499fdd4ddbb5028c2aa6034c73 in term 1.
I20260928 14:58:16.056054 6952 raft_consensus.cc:2454] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 414ed6499fdd4ddbb5028c2aa6034c73 in term 1.
I20260928 14:58:16.056243 7019 leader_election.cc:302] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414ed6499fdd4ddbb5028c2aa6034c73, 467768b6a41440d68ccf709788d81956; no voters:
I20260928 14:58:16.056334 7150 raft_consensus.cc:2781] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:16.056470 7150 raft_consensus.cc:686] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 1 LEADER]: Becoming Leader. State: Replica: 414ed6499fdd4ddbb5028c2aa6034c73, State: Running, Role: LEADER
I20260928 14:58:16.056559 7150 consensus_queue.cc:237] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } }
I20260928 14:58:16.057452 6679 catalog_manager.cc:5495] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 414ed6499fdd4ddbb5028c2aa6034c73 (127.1.252.195). New cstate: current_term: 1 leader_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414ed6499fdd4ddbb5028c2aa6034c73" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 } health_report { overall_health: UNKNOWN } } }
W20260928 14:58:16.064260 6999 tablet.cc:2369] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:16.065747 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.064801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.1.252.195:41041)
0928 14:58:16.064863 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.064864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.064872 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:16.064909 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.064910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.064920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.065006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.065594 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.065595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.065595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.065598 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.065610 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.065614 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.065659 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.065692 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.065702 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:58:16.067340 6952 raft_consensus.cc:1261] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 1 FOLLOWER]: Refusing update from remote peer 414ed6499fdd4ddbb5028c2aa6034c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:16.067392 6821 raft_consensus.cc:1261] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9 [term 1 FOLLOWER]: Refusing update from remote peer 414ed6499fdd4ddbb5028c2aa6034c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:16.067677 7154 consensus_queue.cc:1035] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467768b6a41440d68ccf709788d81956" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:16.067822 7150 consensus_queue.cc:1035] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c5523866ee944a2b90ae085b93e5ec9" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:16.069929 7157 mvcc.cc:205] Tried to move back new op lower bound from 7334328303889559552 to 7334328303848562688. Current Snapshot: MvccSnapshot[applied={T|T < 7334328303889559552}]
I20260928 14:58:16.070516 7158 mvcc.cc:205] Tried to move back new op lower bound from 7334328303889559552 to 7334328303848562688. Current Snapshot: MvccSnapshot[applied={T|T < 7334328303889559552}]
I20260928 14:58:16.072734 6679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:16.072830 6679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:16.073556 6679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ac72969fb684bab9b1247ee226aadef: Sending DeleteTablet for 3 replicas of tablet b4f6d6f524804404bfd9368a8581b4dd
I20260928 14:58:16.073884 6932 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f6d6f524804404bfd9368a8581b4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:16 UTC) from {username='slave'} at 127.0.0.1:51602
I20260928 14:58:16.073978 6801 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f6d6f524804404bfd9368a8581b4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:16 UTC) from {username='slave'} at 127.0.0.1:47006
I20260928 14:58:16.074141 7063 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f6d6f524804404bfd9368a8581b4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:16 UTC) from {username='slave'} at 127.0.0.1:33136
I20260928 14:58:16.074175 7167 tablet_replica.cc:330] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: stopping tablet replica
I20260928 14:58:16.074266 7167 raft_consensus.cc:2229] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:16.074327 7169 tablet_replica.cc:330] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: stopping tablet replica
I20260928 14:58:16.074355 7167 raft_consensus.cc:2258] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:16.074422 7169 raft_consensus.cc:2229] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:16.074556 7168 tablet_replica.cc:330] T b4f6d6f524804404bfd9368a8581b4dd P 2c5523866ee944a2b90ae085b93e5ec9: stopping tablet replica
I20260928 14:58:16.074676 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6739
I20260928 14:58:16.075275 7169 raft_consensus.cc:2258] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:16.075701 7169 ts_tablet_manager.cc:1905] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:16.077579 7169 ts_tablet_manager.cc:1918] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:16.077669 7169 log.cc:1196] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/b4f6d6f524804404bfd9368a8581b4dd
I20260928 14:58:16.077929 7169 ts_tablet_manager.cc:1939] T b4f6d6f524804404bfd9368a8581b4dd P 414ed6499fdd4ddbb5028c2aa6034c73: Deleting consensus metadata
I20260928 14:58:16.078293 6665 catalog_manager.cc:4841] TS 414ed6499fdd4ddbb5028c2aa6034c73 (127.1.252.195:41041): tablet b4f6d6f524804404bfd9368a8581b4dd (table test-table [id=8b4dbc275c8a451794d5d77fac53ba20]) successfully deleted
I20260928 14:58:16.080049 7167 ts_tablet_manager.cc:1905] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:16.081956 7167 ts_tablet_manager.cc:1918] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 14:58:16.081952 6664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:38751 (error 108)
I20260928 14:58:16.082027 7167 log.cc:1196] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/b4f6d6f524804404bfd9368a8581b4dd
I20260928 14:58:16.082300 7167 ts_tablet_manager.cc:1939] T b4f6d6f524804404bfd9368a8581b4dd P 467768b6a41440d68ccf709788d81956: Deleting consensus metadata
W20260928 14:58:16.082458 7140 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:16.082251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:38751 (local address 127.0.0.1:47012)
0928 14:58:16.082284 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.082294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.082333 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.082392 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:38751: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 14:58:16.082581 6664 catalog_manager.cc:4584] TS 2c5523866ee944a2b90ae085b93e5ec9 (127.1.252.193:38751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4f6d6f524804404bfd9368a8581b4dd: Network error: Client connection negotiation failed: client connection to 127.1.252.193:38751: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 14:58:16.082726 6666 catalog_manager.cc:4841] TS 467768b6a41440d68ccf709788d81956 (127.1.252.194:33841): tablet b4f6d6f524804404bfd9368a8581b4dd (table test-table [id=8b4dbc275c8a451794d5d77fac53ba20]) successfully deleted
I20260928 14:58:16.082825 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6870
I20260928 14:58:16.093444 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7001
I20260928 14:58:16.097956 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 6648
2026-09-28T14:58:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (673 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:58:16.117282 2035 mini_oidc.cc:152] Starting JWKS server
I20260928 14:58:16.118340 2035 webserver.cc:460] Webserver started at https://127.0.0.1:42575/ using document root <none> and password file <none>
I20260928 14:58:16.118467 2035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:58:16.118649 2035 webserver.cc:460] Webserver started at http://127.0.0.1:46291/ using document root <none> and password file <none>
2026-09-28T14:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:16Z Disabled control of system clock
I20260928 14:58:16.126403 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41139
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41139
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/testchainca.cert with env {}
W20260928 14:58:16.207358 7176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:16.207554 7176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:16.207576 7176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:16.208736 7176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:16.208777 7176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:16.208791 7176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:16.208802 7176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:16.208814 7176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:58:16.208827 7176 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:58:16.208838 7176 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42575/jwks.json
I20260928 14:58:16.210332 7176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:16.210515 7176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:16.210661 7176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:16.215333 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:16.215298 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.215324 7176 server_base.cc:1034] running on GCE node
W20260928 14:58:16.215298 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.215787 7176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:16.216094 7176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:16.217242 7176 hybrid_clock.cc:648] HybridClock initialized: now 1790607496217232 us; error 31 us; skew 500 ppm
I20260928 14:58:16.218328 7176 webserver.cc:460] Webserver started at http://127.1.252.254:37851/ using document root <none> and password file <none>
I20260928 14:58:16.218533 7176 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:16.218575 7176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:16.218657 7176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:16.219594 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "00b1b6dcd759414b94f4c171db366979"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "4ae57263ee71e464912214b7404cf226"
server_key_iv: "9a3d4a3f4144d8615915a1fa79a805bc"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.219877 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "00b1b6dcd759414b94f4c171db366979"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "4ae57263ee71e464912214b7404cf226"
server_key_iv: "9a3d4a3f4144d8615915a1fa79a805bc"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.221117 7176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:16.221778 7190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:16.221930 7176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:16.222005 7176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "00b1b6dcd759414b94f4c171db366979"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "4ae57263ee71e464912214b7404cf226"
server_key_iv: "9a3d4a3f4144d8615915a1fa79a805bc"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.222056 7176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:16.234282 7176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:16.234531 7176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:16.234627 7176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:16.238327 7176 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:41139
I20260928 14:58:16.238379 7242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:41139 every 8 connection(s)
I20260928 14:58:16.238719 7176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:16.239661 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:16.240607 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7176
I20260928 14:58:16.240695 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:16.240871 2035 external_mini_cluster.cc:1428] Setting key 60cf5849c45bce4ebb083e9d6a66d80c
I20260928 14:58:16.242600 7243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979: Bootstrap starting.
I20260928 14:58:16.243331 7243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:16.243643 7243 log.cc:824] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:16.244427 7243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979: No bootstrap required, opened a new log
I20260928 14:58:16.246680 7243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1b6dcd759414b94f4c171db366979" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41139 } }
I20260928 14:58:16.246858 7243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:16.246904 7243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b1b6dcd759414b94f4c171db366979, State: Initialized, Role: FOLLOWER
I20260928 14:58:16.247040 7243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1b6dcd759414b94f4c171db366979" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41139 } }
I20260928 14:58:16.247136 7243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:16.247192 7243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:16.247239 7243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.247294 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.241430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.1.252.254:41139)
0928 14:58:16.241637 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.241647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.241679 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:58:16.242320 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.242328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.242434 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.242587 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.242787 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.244214 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.245033 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.245665 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.245794 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.246437 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.246441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.246442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.246650 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.246692 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.246708 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.246776 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.246853 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.247030 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 14:58:16.247874 7243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1b6dcd759414b94f4c171db366979" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41139 } }
I20260928 14:58:16.248010 7243 leader_election.cc:302] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00b1b6dcd759414b94f4c171db366979; no voters:
I20260928 14:58:16.248173 7243 leader_election.cc:288] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:16.248245 7248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:16.248314 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:58:16.248406 7248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [term 1 LEADER]: Becoming Leader. State: Replica: 00b1b6dcd759414b94f4c171db366979, State: Running, Role: LEADER
I20260928 14:58:16.248441 7243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:16.248512 7248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1b6dcd759414b94f4c171db366979" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41139 } }
I20260928 14:58:16.248833 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00b1b6dcd759414b94f4c171db366979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1b6dcd759414b94f4c171db366979" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41139 } } }
I20260928 14:58:16.248922 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:16.249053 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00b1b6dcd759414b94f4c171db366979. Latest consensus state: current_term: 1 leader_uuid: "00b1b6dcd759414b94f4c171db366979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1b6dcd759414b94f4c171db366979" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41139 } } }
I20260928 14:58:16.249115 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979 [sys.catalog]: This master's current role is: LEADER
W20260928 14:58:16.250205 7264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:58:16.250268 7264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:58:16.250342 7255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:16.250542 7255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:16.251986 7255 catalog_manager.cc:1347] Generated new cluster ID: c2275305f0f24b808a2f73988178dd3b
I20260928 14:58:16.252040 7255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:16.268460 7255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:16.269127 7255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:16.272971 7255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00b1b6dcd759414b94f4c171db366979: Generated new TSK 0
I20260928 14:58:16.273137 7255 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:58:16.298998 7273 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:58:16.299100 7273 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:58:16.299329 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.284479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.1.252.254:41139)
0928 14:58:16.284545 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.284547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.284685 (+ 138us) server_negotiation.cc:407] Connection header received
0928 14:58:16.284786 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.284787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.284796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.284847 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:58:16.284994 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.285474 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.286584 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.286768 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.286815 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.287032 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:58:16.299115 (+ 12083us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:58:16.299268 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 14:58:16.299408 7246 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 14:58:16.299674 7272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:16.284427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41139 (local address 127.0.0.1:33922)
0928 14:58:16.284612 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.284623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.284695 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.284860 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:16.284861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:16.284863 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:58:16.284935 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.284938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.285552 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.285553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.286502 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.286506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.286841 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.286842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.286910 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.286932 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:58:16.299271 (+ 12339us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:58:16.299389 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.252.254:41139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.252.254:41139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 14:58:16.300134 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7176
2026-09-28T14:58:16Z chronyd exiting
I20260928 14:58:16.336158 2035 test_util.cc:175] -----------------------------------------------
I20260928 14:58:16.336233 2035 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607474232031-2035-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:58:16.337039 2035 mini_oidc.cc:152] Starting JWKS server
I20260928 14:58:16.337741 2035 webserver.cc:460] Webserver started at https://127.0.0.1:43723/ using document root <none> and password file <none>
I20260928 14:58:16.337781 2035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:58:16.337944 2035 webserver.cc:460] Webserver started at http://127.0.0.1:32893/ using document root <none> and password file <none>
2026-09-28T14:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:16Z Disabled control of system clock
I20260928 14:58:16.345652 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42219
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:42219
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/testchainca.cert with env {}
W20260928 14:58:16.426018 7279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:16.426213 7279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:16.426235 7279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:16.427294 7279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:16.427328 7279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:16.427341 7279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:16.427353 7279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:16.427366 7279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:58:16.427378 7279 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:58:16.427389 7279 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43723/jwks.json
I20260928 14:58:16.428850 7279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:42219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:16.429051 7279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:16.429214 7279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:16.433856 7284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.433943 7279 server_base.cc:1034] running on GCE node
W20260928 14:58:16.433853 7287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:16.433868 7285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.434366 7279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:16.434602 7279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:16.435743 7279 hybrid_clock.cc:648] HybridClock initialized: now 1790607496435731 us; error 32 us; skew 500 ppm
I20260928 14:58:16.437021 7279 webserver.cc:460] Webserver started at http://127.1.252.254:46323/ using document root <none> and password file <none>
I20260928 14:58:16.437255 7279 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:16.437300 7279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:16.437487 7279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:16.438493 7279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "fbb4a1fb6e6e72bb0d83fb5e59263a78"
server_key_iv: "7b3c9c1fcab7f7519fa4ed75fb839652"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.438850 7279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "fbb4a1fb6e6e72bb0d83fb5e59263a78"
server_key_iv: "7b3c9c1fcab7f7519fa4ed75fb839652"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.440171 7279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:16.440882 7293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:16.441059 7279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:16.441160 7279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "fbb4a1fb6e6e72bb0d83fb5e59263a78"
server_key_iv: "7b3c9c1fcab7f7519fa4ed75fb839652"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.441232 7279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:16.453742 7279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:16.454032 7279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:16.454178 7279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:16.458112 7279 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:42219
I20260928 14:58:16.458141 7345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:42219 every 8 connection(s)
I20260928 14:58:16.458480 7279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:16.459421 7346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:16.459990 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7279
I20260928 14:58:16.460084 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:16.460268 2035 external_mini_cluster.cc:1428] Setting key d19e8bd14444589127a9d174730c1052
I20260928 14:58:16.462060 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1: Bootstrap starting.
I20260928 14:58:16.462643 7346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:16.463008 7346 log.cc:824] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:16.463735 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1: No bootstrap required, opened a new log
I20260928 14:58:16.465758 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.460770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.1.252.254:42219)
0928 14:58:16.460982 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.460990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.461015 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:58:16.461405 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.461411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.461468 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.461614 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.461804 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.462710 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.463824 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.464303 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.464395 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.465100 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.465103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.465104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.465262 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.465293 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.465303 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.465354 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.465467 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.465527 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260928 14:58:16.466015 7346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42219 } }
I20260928 14:58:16.466223 7346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:16.466296 7346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023589d74d1b4f7bbf0fbaf821c9ecf1, State: Initialized, Role: FOLLOWER
I20260928 14:58:16.466424 7346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42219 } }
I20260928 14:58:16.466513 7346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:16.466537 7346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:16.466607 7346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.466866 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:58:16.467254 7346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42219 } }
I20260928 14:58:16.467371 7346 leader_election.cc:302] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 023589d74d1b4f7bbf0fbaf821c9ecf1; no voters:
I20260928 14:58:16.467551 7346 leader_election.cc:288] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:16.467720 7352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:16.467813 7346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:16.467832 7352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [term 1 LEADER]: Becoming Leader. State: Replica: 023589d74d1b4f7bbf0fbaf821c9ecf1, State: Running, Role: LEADER
I20260928 14:58:16.467959 7352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42219 } }
I20260928 14:58:16.468291 7352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023589d74d1b4f7bbf0fbaf821c9ecf1. Latest consensus state: current_term: 1 leader_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42219 } } }
I20260928 14:58:16.468394 7352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:16.468307 7351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023589d74d1b4f7bbf0fbaf821c9ecf1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42219 } } }
I20260928 14:58:16.468497 7351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:16.468600 7360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:16.469105 7360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:16.470546 7360 catalog_manager.cc:1347] Generated new cluster ID: 246a5a676dad44699f65434e91118e22
I20260928 14:58:16.470613 7360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:16.492245 7360 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:16.492774 7360 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:16.497166 7360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 023589d74d1b4f7bbf0fbaf821c9ecf1: Generated new TSK 0
I20260928 14:58:16.497349 7360 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:58:16.524897 7376 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:58:16.524986 7376 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:58:16.525223 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.503565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.1.252.254:42219)
0928 14:58:16.503632 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.503635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.503768 (+ 133us) server_negotiation.cc:407] Connection header received
0928 14:58:16.503870 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.503871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.503917 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.503980 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:58:16.504111 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.504609 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.505387 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.505597 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.505673 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.505856 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:58:16.525021 (+ 19165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:58:16.525171 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:58:16.525297 7349 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 14:58:16.525377 7375 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:16.503500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42219 (local address 127.0.0.1:48890)
0928 14:58:16.503687 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.503697 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.503780 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.503977 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:16.503978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:16.503980 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:58:16.504040 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.504042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.504695 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.504696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.505297 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.505299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.505675 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.505676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.505738 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.505742 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:58:16.525136 (+ 19394us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:58:16.525272 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.252.254:42219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260928 14:58:16.525702 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7279
2026-09-28T14:58:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:58:16.563309 2035 mini_oidc.cc:152] Starting JWKS server
I20260928 14:58:16.564025 2035 webserver.cc:460] Webserver started at https://127.0.0.1:39205/ using document root <none> and password file <none>
I20260928 14:58:16.564071 2035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:58:16.564182 2035 webserver.cc:460] Webserver started at http://127.0.0.1:38229/ using document root <none> and password file <none>
2026-09-28T14:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:16Z Disabled control of system clock
I20260928 14:58:16.571647 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41449
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39205/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:58:16.650221 7382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:16.650424 7382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:16.650444 7382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:16.651513 7382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:16.651546 7382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:16.651558 7382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:16.651571 7382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:16.651582 7382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:58:16.651594 7382 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:58:16.651605 7382 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39205/jwks.json
I20260928 14:58:16.652875 7382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39205/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:16.653077 7382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:16.653286 7382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:16.658042 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:16.658120 7388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.658149 7382 server_base.cc:1034] running on GCE node
W20260928 14:58:16.658043 7387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.658496 7382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:16.658721 7382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:16.659948 7382 hybrid_clock.cc:648] HybridClock initialized: now 1790607496659933 us; error 32 us; skew 500 ppm
I20260928 14:58:16.661098 7382 webserver.cc:460] Webserver started at http://127.1.252.254:34295/ using document root <none> and password file <none>
I20260928 14:58:16.661319 7382 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:16.661363 7382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:16.661535 7382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:16.662550 7382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "528f7da3b95847ce905f62ea7dc6a50a"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "7c926995f0bb728aeffbbd97b4bdf089"
server_key_iv: "3cab7da57a0f30635eed4dc207ab46f2"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.662899 7382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "528f7da3b95847ce905f62ea7dc6a50a"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "7c926995f0bb728aeffbbd97b4bdf089"
server_key_iv: "3cab7da57a0f30635eed4dc207ab46f2"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.664175 7382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:16.664927 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:16.665115 7382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:16.665203 7382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "528f7da3b95847ce905f62ea7dc6a50a"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "7c926995f0bb728aeffbbd97b4bdf089"
server_key_iv: "3cab7da57a0f30635eed4dc207ab46f2"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.665287 7382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:16.678696 7382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:16.678928 7382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:16.679028 7382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:16.682955 7382 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:41449
I20260928 14:58:16.682986 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:41449 every 8 connection(s)
I20260928 14:58:16.683315 7382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:16.684036 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:16.685571 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7382
I20260928 14:58:16.685652 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:16.685823 2035 external_mini_cluster.cc:1428] Setting key 56b843bfda9158a0c5d197bd9e97daa3
I20260928 14:58:16.687139 7449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a: Bootstrap starting.
I20260928 14:58:16.687839 7449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:16.688189 7449 log.cc:824] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:16.688920 7449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a: No bootstrap required, opened a new log
I20260928 14:58:16.691095 7449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f7da3b95847ce905f62ea7dc6a50a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41449 } }
I20260928 14:58:16.691272 7449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:16.691310 7449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528f7da3b95847ce905f62ea7dc6a50a, State: Initialized, Role: FOLLOWER
I20260928 14:58:16.691457 7449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f7da3b95847ce905f62ea7dc6a50a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41449 } }
I20260928 14:58:16.691555 7449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:16.691589 7449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:16.691637 7449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.691880 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.686313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.1.252.254:41449)
0928 14:58:16.686524 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.686533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.686568 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:58:16.687118 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.687126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.687206 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.687360 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.687525 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.688843 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.689642 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.690282 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.690419 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.691022 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.691026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.691026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.691227 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.691271 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.691286 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.691539 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.691573 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.691655 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260928 14:58:16.692335 7449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f7da3b95847ce905f62ea7dc6a50a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41449 } }
I20260928 14:58:16.692472 7449 leader_election.cc:302] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528f7da3b95847ce905f62ea7dc6a50a; no voters:
I20260928 14:58:16.692646 7449 leader_election.cc:288] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:16.692672 7454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:16.692871 7454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [term 1 LEADER]: Becoming Leader. State: Replica: 528f7da3b95847ce905f62ea7dc6a50a, State: Running, Role: LEADER
I20260928 14:58:16.692862 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:58:16.692888 7449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:16.693065 7454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f7da3b95847ce905f62ea7dc6a50a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41449 } }
I20260928 14:58:16.694341 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528f7da3b95847ce905f62ea7dc6a50a. Latest consensus state: current_term: 1 leader_uuid: "528f7da3b95847ce905f62ea7dc6a50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f7da3b95847ce905f62ea7dc6a50a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41449 } } }
I20260928 14:58:16.694346 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528f7da3b95847ce905f62ea7dc6a50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528f7da3b95847ce905f62ea7dc6a50a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41449 } } }
I20260928 14:58:16.694660 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:16.694696 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a [sys.catalog]: This master's current role is: LEADER
W20260928 14:58:16.694803 7469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:58:16.694861 7469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:58:16.694924 7471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:16.695078 7471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:16.696362 7471 catalog_manager.cc:1347] Generated new cluster ID: b2e4dfe53b1f486681a903a9b7f883b8
I20260928 14:58:16.696421 7471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:16.713145 7471 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:16.713845 7471 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:16.723542 7471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528f7da3b95847ce905f62ea7dc6a50a: Generated new TSK 0
I20260928 14:58:16.723753 7471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:16.736574 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.733265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.1.252.254:41449)
0928 14:58:16.733326 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.733328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.733450 (+ 122us) server_negotiation.cc:407] Connection header received
0928 14:58:16.733529 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.733530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.733541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.733623 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:58:16.733765 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.734249 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.735068 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.735283 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.735360 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.735539 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:58:16.736347 (+ 808us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:58:16.736497 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:58:16.736630 7452 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 14:58:16.736665 7478 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:16.733215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41449 (local address 127.0.0.1:48742)
0928 14:58:16.733375 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.733385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.733468 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.733623 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:16.733624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:16.733626 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:58:16.733688 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.733690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.734352 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.734353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.734981 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.734983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.735364 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.735366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.735426 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.735431 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:58:16.736470 (+ 1039us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:58:16.736557 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.252.254:41449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39205/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260928 14:58:16.736881 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7382
E20260928 14:58:16.748059 7479 webserver.cc:533] Webserver: sslize failed:
E20260928 14:58:16.748121 7479 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T14:58:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:16Z Disabled control of system clock
I20260928 14:58:16.781912 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:39691
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:39691
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/testchainca.cert with env {}
W20260928 14:58:16.861268 7483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:16.861497 7483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:16.861536 7483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:16.862706 7483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:16.862773 7483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:16.862794 7483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:16.862815 7483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:16.862843 7483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:58:16.862871 7483 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:58:16.862893 7483 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:58:16.864462 7483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:39691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:16.864735 7483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:16.864959 7483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:16.870107 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:16.870100 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.870162 7483 server_base.cc:1034] running on GCE node
W20260928 14:58:16.870117 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:16.870615 7483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:16.870852 7483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:16.871985 7483 hybrid_clock.cc:648] HybridClock initialized: now 1790607496871949 us; error 48 us; skew 500 ppm
I20260928 14:58:16.873329 7483 webserver.cc:460] Webserver started at http://127.1.252.254:46323/ using document root <none> and password file <none>
I20260928 14:58:16.873628 7483 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:16.873740 7483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:16.873898 7483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:16.874958 7483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "8868946590664c0a88c3ac632e2c75e6"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "06da6dd7666aea5bacd307c8fb28b786"
server_key_iv: "82ae64442bedc31b4e52fe436792926b"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.875315 7483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "8868946590664c0a88c3ac632e2c75e6"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "06da6dd7666aea5bacd307c8fb28b786"
server_key_iv: "82ae64442bedc31b4e52fe436792926b"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.876685 7483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:16.877427 7497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:16.877615 7483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:16.877717 7483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "8868946590664c0a88c3ac632e2c75e6"
format_stamp: "Formatted at 2026-09-28 14:58:16 on dist-test-slave-sf70"
server_key: "06da6dd7666aea5bacd307c8fb28b786"
server_key_iv: "82ae64442bedc31b4e52fe436792926b"
server_key_version: "encryptionkey@0"
I20260928 14:58:16.877800 7483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:16.898239 7483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:16.898511 7483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:16.898617 7483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:16.902362 7483 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:39691
I20260928 14:58:16.902439 7549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:39691 every 8 connection(s)
I20260928 14:58:16.902757 7483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:16.903828 7550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:16.906194 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7483
I20260928 14:58:16.906320 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:16.906402 7550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6: Bootstrap starting.
I20260928 14:58:16.906508 2035 external_mini_cluster.cc:1428] Setting key 2cf047fd4c40c07186f92de2d1029dac
I20260928 14:58:16.907121 7550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:16.907401 7550 log.cc:824] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:16.908125 7550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6: No bootstrap required, opened a new log
I20260928 14:58:16.909902 7550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8868946590664c0a88c3ac632e2c75e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39691 } }
I20260928 14:58:16.910055 7550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:16.910102 7550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8868946590664c0a88c3ac632e2c75e6, State: Initialized, Role: FOLLOWER
I20260928 14:58:16.910238 7550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8868946590664c0a88c3ac632e2c75e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39691 } }
I20260928 14:58:16.910342 7550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:16.910388 7550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:16.910436 7550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:16.910955 7550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8868946590664c0a88c3ac632e2c75e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39691 } }
I20260928 14:58:16.911070 7550 leader_election.cc:302] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8868946590664c0a88c3ac632e2c75e6; no voters:
I20260928 14:58:16.911237 7550 leader_election.cc:288] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:16.911278 7555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:16.911417 7555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8868946590664c0a88c3ac632e2c75e6, State: Running, Role: LEADER
I20260928 14:58:16.911507 7550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:16.911512 7555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8868946590664c0a88c3ac632e2c75e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39691 } }
I20260928 14:58:16.911847 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8868946590664c0a88c3ac632e2c75e6. Latest consensus state: current_term: 1 leader_uuid: "8868946590664c0a88c3ac632e2c75e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8868946590664c0a88c3ac632e2c75e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39691 } } }
I20260928 14:58:16.911966 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:16.911867 7557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8868946590664c0a88c3ac632e2c75e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8868946590664c0a88c3ac632e2c75e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39691 } } }
I20260928 14:58:16.912364 7557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:16.912539 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.907059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.1.252.254:39691)
0928 14:58:16.907284 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.907293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.907329 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:58:16.907838 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.907844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.907962 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.908107 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:16.908301 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.909192 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.909992 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.910578 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.910712 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.911493 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:16.911497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:16.911497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:16.911735 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:16.911777 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:16.911791 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:16.911856 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:16.912156 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 14:58:16.912266 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260928 14:58:16.912717 7560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:16.913362 7560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:16.913882 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:58:16.915192 7560 catalog_manager.cc:1347] Generated new cluster ID: 9b4fce975aaf4e8c904c7da9659e1ed5
I20260928 14:58:16.915253 7560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:16.939284 7560 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:16.940062 7560 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:16.946352 7560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8868946590664c0a88c3ac632e2c75e6: Generated new TSK 0
I20260928 14:58:16.946559 7560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:17.424307 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:16.960265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.1.252.254:39691)
0928 14:58:16.960329 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:16.960332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:16.960468 (+ 136us) server_negotiation.cc:407] Connection header received
0928 14:58:16.960541 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:16.960542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:16.960553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:16.960608 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:58:16.960765 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.961247 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.962397 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.962599 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.962646 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.962842 (+ 196us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:58:17.424071 (+461229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:58:17.424243 (+ 172us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36794: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 14:58:17.424402 7553 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36794: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 14:58:17.424474 7579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:16.960176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:39691 (local address 127.0.0.1:36794)
0928 14:58:16.960394 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:16.960404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:16.960471 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:16.960627 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:16.960628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:16.960630 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:58:16.960696 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.960699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.961347 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.961348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.962308 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:16.962311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:16.962672 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:16.962673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:16.962736 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:16.962741 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:58:17.424235 (+461494us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:58:17.424354 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.252.254:39691: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
I20260928 14:58:17.424829 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7483
2026-09-28T14:58:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (668 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:17 dist-test-slave-sf70 krb5kdc[7591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:17 dist-test-slave-sf70 krb5kdc[7591](info): set up 2 sockets
Sep 28 14:58:17 dist-test-slave-sf70 krb5kdc[7591](info): commencing operation
krb5kdc: starting...
W20260928 14:58:18.641754 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:18 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:18.759012 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:43295
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:18.837903 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:18.838097 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:18.838119 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:18.839265 7604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:18.839306 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:18.839320 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:18.839334 7604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:18.839385 7604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:18.840831 7604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:18.841085 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:18.841301 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:18.845973 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:18.846035 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:18.846037 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:18.846172 7604 server_base.cc:1034] running on GCE node
I20260928 14:58:18.846405 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:18.846633 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:18.847787 7604 hybrid_clock.cc:648] HybridClock initialized: now 1790607498847756 us; error 42 us; skew 500 ppm
Sep 28 14:58:18 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607498, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:18.850160 7604 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:18.850589 7604 webserver.cc:460] Webserver started at http://127.1.252.254:42499/ using document root <none> and password file <none>
I20260928 14:58:18.850760 7604 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:18.850800 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:18.850885 7604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:18.851874 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "eb5c2e58f09a43ef84104e855f005af4"
format_stamp: "Formatted at 2026-09-28 14:58:18 on dist-test-slave-sf70"
server_key: "8efeae44f9c30e593454cd00abdd1563"
server_key_iv: "83179a89b6df5de6c9485894c6bfbdad"
server_key_version: "encryptionkey@0"
I20260928 14:58:18.852295 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "eb5c2e58f09a43ef84104e855f005af4"
format_stamp: "Formatted at 2026-09-28 14:58:18 on dist-test-slave-sf70"
server_key: "8efeae44f9c30e593454cd00abdd1563"
server_key_iv: "83179a89b6df5de6c9485894c6bfbdad"
server_key_version: "encryptionkey@0"
I20260928 14:58:18.853487 7604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:18.854103 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:18.854266 7604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:18.854365 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "eb5c2e58f09a43ef84104e855f005af4"
format_stamp: "Formatted at 2026-09-28 14:58:18 on dist-test-slave-sf70"
server_key: "8efeae44f9c30e593454cd00abdd1563"
server_key_iv: "83179a89b6df5de6c9485894c6bfbdad"
server_key_version: "encryptionkey@0"
I20260928 14:58:18.854441 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:18.897469 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:18.899380 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:18.899542 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:18.903597 7604 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:43295
I20260928 14:58:18.903672 7671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:43295 every 8 connection(s)
I20260928 14:58:18.904034 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:18.904924 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:18.907256 7672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4: Bootstrap starting.
I20260928 14:58:18.907837 7672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:18.908177 7672 log.cc:824] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:18.908886 7672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4: No bootstrap required, opened a new log
I20260928 14:58:18.910490 7672 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5c2e58f09a43ef84104e855f005af4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43295 } }
I20260928 14:58:18.910622 7672 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:18.910645 7672 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5c2e58f09a43ef84104e855f005af4, State: Initialized, Role: FOLLOWER
I20260928 14:58:18.910823 7672 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5c2e58f09a43ef84104e855f005af4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43295 } }
I20260928 14:58:18.910888 7672 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:18.910909 7672 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:18.910976 7672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:18.911509 7672 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5c2e58f09a43ef84104e855f005af4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43295 } }
I20260928 14:58:18.911616 7672 leader_election.cc:302] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5c2e58f09a43ef84104e855f005af4; no voters:
I20260928 14:58:18.911808 7672 leader_election.cc:288] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:18.911909 7675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:18.912075 7672 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:18.912302 7675 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [term 1 LEADER]: Becoming Leader. State: Replica: eb5c2e58f09a43ef84104e855f005af4, State: Running, Role: LEADER
I20260928 14:58:18.912432 7675 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5c2e58f09a43ef84104e855f005af4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43295 } }
I20260928 14:58:18.912840 7676 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb5c2e58f09a43ef84104e855f005af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5c2e58f09a43ef84104e855f005af4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43295 } } }
I20260928 14:58:18.912878 7677 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5c2e58f09a43ef84104e855f005af4. Latest consensus state: current_term: 1 leader_uuid: "eb5c2e58f09a43ef84104e855f005af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5c2e58f09a43ef84104e855f005af4" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43295 } } }
I20260928 14:58:18.912935 7676 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:18.912958 7677 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:18.913182 7684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:18.913717 7684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:18.913803 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7604
I20260928 14:58:18.913908 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:18.914101 2035 external_mini_cluster.cc:1428] Setting key a4d4846ed3e924731e7ee72a81f73f49
I20260928 14:58:18.915350 7684 catalog_manager.cc:1347] Generated new cluster ID: e097d41551f54b4aa9def2b180eb7a45
I20260928 14:58:18.915410 7684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:18 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:18.925836 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:18.914590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.1.252.254:43295)
0928 14:58:18.914752 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:18.914761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:18.914959 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:58:18.915148 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:18.915156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:18.915221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:18.915363 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:18.916110 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:18.917552 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:18.918584 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:18.919206 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:18.919319 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:18.922126 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:18.922130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:18.922131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:18.922142 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:18.924068 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:18.924469 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:18.924471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:18.924472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:18.924572 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:18.924777 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:18.924778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:18.924778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:18.925074 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:18.925210 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:18.925336 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 14:58:18.925565 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 14:58:18.935938 7684 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:18.936650 7684 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:18.941797 7684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb5c2e58f09a43ef84104e855f005af4: Generated new TSK 0
I20260928 14:58:18.941978 7684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:18.970460 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43295
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:19.050212 7698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:19.050493 7698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:19.050582 7698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:19.051723 7698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:19.051788 7698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:19.051831 7698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:19.053344 7698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:19.053606 7698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:19.053828 7698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:19.059089 7704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:19.059219 7698 server_base.cc:1034] running on GCE node
W20260928 14:58:19.059069 7703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:19.059065 7706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:19.059592 7698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:19.059808 7698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:19.060906 7698 hybrid_clock.cc:648] HybridClock initialized: now 1790607499060891 us; error 28 us; skew 500 ppm
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.063386 7698 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:19.063814 7698 webserver.cc:460] Webserver started at http://127.1.252.193:38119/ using document root <none> and password file <none>
I20260928 14:58:19.064069 7698 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:19.064114 7698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:19.064257 7698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:19.065318 7698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "03f8cfff2965468a8bb6b9429ad26acb"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "e1f299299fc67290658da742aa28826d"
server_key_iv: "bf0d9e54b465639456742f5576daa746"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.065675 7698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "03f8cfff2965468a8bb6b9429ad26acb"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "e1f299299fc67290658da742aa28826d"
server_key_iv: "bf0d9e54b465639456742f5576daa746"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.066995 7698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:19.067680 7713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.067852 7698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:19.067979 7698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "03f8cfff2965468a8bb6b9429ad26acb"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "e1f299299fc67290658da742aa28826d"
server_key_iv: "bf0d9e54b465639456742f5576daa746"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.068053 7698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:19.076546 7698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:19.078531 7698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:19.078701 7698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:19.078923 7698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:19.079222 7698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:19.079277 7698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.079317 7698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:19.079344 7698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.085206 7698 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:37739
I20260928 14:58:19.085258 7826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:37739 every 8 connection(s)
I20260928 14:58:19.085548 7698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:19.094345 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.086209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43021 (local address 127.1.252.254:43295)
0928 14:58:19.086251 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.086254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.086463 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:58:19.087400 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.087402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.087417 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.087476 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.088169 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.088694 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.089636 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.089837 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.089891 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.091822 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:19.091823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:19.091824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:19.091832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:19.093265 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.093620 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.093621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.093621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.093670 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.093866 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.093866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.093867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.093962 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:19.094011 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:19.094223 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:58:19.094284 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:19.094549 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.086171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43295 (local address 127.1.252.193:43021)
0928 14:58:19.086329 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:19.086369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:19.087283 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:19.087556 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:19.087559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:19.087801 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:19.088078 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:19.088083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.088772 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.088772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:19.089557 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:19.089559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.089909 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.089910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:19.090005 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.090439 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:19.090447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:19.091735 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:19.093333 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:19.093340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:19.093341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:19.093555 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:19.093724 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:19.093725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:19.093725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:19.093814 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:19.094013 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:19.094017 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:19.094118 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 14:58:19.094194 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:58:19.094304 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260928 14:58:19.095050 7827 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43295
I20260928 14:58:19.095151 7827 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:19.095346 7827 heartbeater.cc:511] Master 127.1.252.254:43295 requested a full tablet report, sending...
I20260928 14:58:19.095363 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7698
I20260928 14:58:19.095485 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:19.095714 2035 external_mini_cluster.cc:1428] Setting key cbd8b303b5ec58ba4fa78d688002a847
I20260928 14:58:19.095938 7636 ts_manager.cc:194] Registered new tserver with Master: 03f8cfff2965468a8bb6b9429ad26acb (127.1.252.193:37739)
I20260928 14:58:19.096778 7636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:43021
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:19.124333 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43295
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:19.205041 7832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:19.205226 7832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:19.205248 7832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:19.206312 7832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:19.206346 7832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:19.206370 7832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:19.207726 7832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:19.207967 7832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:19.208149 7832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:19.213120 7837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:19.213317 7832 server_base.cc:1034] running on GCE node
W20260928 14:58:19.213361 7838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:19.213357 7840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:19.213635 7832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:19.213915 7832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:19.215063 7832 hybrid_clock.cc:648] HybridClock initialized: now 1790607499215057 us; error 54 us; skew 500 ppm
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.217518 7832 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:19.217989 7832 webserver.cc:460] Webserver started at http://127.1.252.194:44231/ using document root <none> and password file <none>
I20260928 14:58:19.218220 7832 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:19.218287 7832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:19.218454 7832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:19.219496 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "88bc32ecf290489c8a453aae57ec2e3c"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "358fb8ca59c89605604571c2ef9f83c2"
server_key_iv: "fcdd93248e828818791cdee937d6121f"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.219851 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "88bc32ecf290489c8a453aae57ec2e3c"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "358fb8ca59c89605604571c2ef9f83c2"
server_key_iv: "fcdd93248e828818791cdee937d6121f"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.221238 7832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:19.222021 7847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.222210 7832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:19.222304 7832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "88bc32ecf290489c8a453aae57ec2e3c"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "358fb8ca59c89605604571c2ef9f83c2"
server_key_iv: "fcdd93248e828818791cdee937d6121f"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.222383 7832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:19.235229 7832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:19.237007 7832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:19.237161 7832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:19.237373 7832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:19.237666 7832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:19.237718 7832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.237758 7832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:19.237785 7832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.243773 7832 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:46675
I20260928 14:58:19.243829 7960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:46675 every 8 connection(s)
I20260928 14:58:19.244171 7832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:19.248903 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7832
I20260928 14:58:19.249073 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:19.249322 2035 external_mini_cluster.cc:1428] Setting key 1fa592e073e2bc2f4a6f5be8c5b5a9e8
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:19.254449 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.244900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:60801 (local address 127.1.252.254:43295)
0928 14:58:19.244967 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.244969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.245195 (+ 226us) server_negotiation.cc:407] Connection header received
0928 14:58:19.246164 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.246165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.246179 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.246230 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.246811 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.247341 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.248469 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.248686 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.248772 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.251519 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:19.251520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:19.251521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:19.251529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:19.253072 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.253482 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.253483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.253484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.253546 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.253808 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.253809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.253810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.253927 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:19.254003 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:19.254268 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:58:19.254361 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:58:19.254637 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.244906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43295 (local address 127.1.252.194:60801)
0928 14:58:19.245081 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:19.245116 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:19.246049 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:19.246293 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:19.246297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:19.246541 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:19.246720 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:19.246724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.247441 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.247442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:19.248366 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:19.248369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.248783 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.248784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:19.248919 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.249545 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:19.249555 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:19.251424 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:19.253164 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:19.253172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:19.253174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:19.253397 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:19.253611 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:19.253612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:19.253612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:19.253747 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:19.253994 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:19.254000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:19.254138 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:58:19.254247 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:58:19.254406 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260928 14:58:19.255153 7961 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43295
I20260928 14:58:19.255268 7961 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:19.255452 7961 heartbeater.cc:511] Master 127.1.252.254:43295 requested a full tablet report, sending...
I20260928 14:58:19.255934 7636 ts_manager.cc:194] Registered new tserver with Master: 88bc32ecf290489c8a453aae57ec2e3c (127.1.252.194:46675)
I20260928 14:58:19.256309 7636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:60801
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:19.275516 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43295
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:19.354765 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:19.354960 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:19.354982 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:19.356061 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:19.356097 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:19.356122 7966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:19.357810 7966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:19.358055 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:19.358286 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:19.363202 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:19.363209 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:19.363209 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:19.363579 7966 server_base.cc:1034] running on GCE node
I20260928 14:58:19.363807 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:19.364086 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:19.365248 7966 hybrid_clock.cc:648] HybridClock initialized: now 1790607499365235 us; error 34 us; skew 500 ppm
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.367726 7966 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:19.368270 7966 webserver.cc:460] Webserver started at http://127.1.252.195:43799/ using document root <none> and password file <none>
I20260928 14:58:19.368521 7966 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:19.368587 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:19.368717 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:19.369727 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "b51769f43e914e37ba3432f5aa3cfb1f"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "f24de74278fbee189971716df7480461"
server_key_iv: "05dd304faa0d3a78683ae54c75c567e1"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.370075 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "b51769f43e914e37ba3432f5aa3cfb1f"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "f24de74278fbee189971716df7480461"
server_key_iv: "05dd304faa0d3a78683ae54c75c567e1"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.371419 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:19.372187 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.372402 7966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:58:19.372488 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "b51769f43e914e37ba3432f5aa3cfb1f"
format_stamp: "Formatted at 2026-09-28 14:58:19 on dist-test-slave-sf70"
server_key: "f24de74278fbee189971716df7480461"
server_key_iv: "05dd304faa0d3a78683ae54c75c567e1"
server_key_version: "encryptionkey@0"
I20260928 14:58:19.372541 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:19.382194 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:19.384337 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:19.384443 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:19.384634 7966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:19.384907 7966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:19.384939 7966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.384959 7966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:19.384972 7966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:19.390758 7966 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:38679
I20260928 14:58:19.390805 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:38679 every 8 connection(s)
I20260928 14:58:19.391100 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:19.399586 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.391752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:60343 (local address 127.1.252.254:43295)
0928 14:58:19.391814 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.391823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.392099 (+ 276us) server_negotiation.cc:407] Connection header received
0928 14:58:19.392982 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.392983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.392993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.393040 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.393583 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.394072 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.395006 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.395185 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.395235 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.397293 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:19.397294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:19.397294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:19.397300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:19.398434 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.398773 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.398773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.398774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.398821 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.399077 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.399078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.399078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.399177 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:19.399227 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:19.399464 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 14:58:19.399523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:58:19.399791 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.391736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43295 (local address 127.1.252.195:60343)
0928 14:58:19.391967 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:19.392011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:19.392893 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:19.393097 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:19.393101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:19.393333 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:19.393513 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:19.393517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.394149 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.394149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:19.394930 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:19.394933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.395254 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.395255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:19.395349 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.395768 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:19.395776 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:19.397208 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:19.398502 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:19.398507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:19.398508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:19.398711 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:19.398905 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:19.398906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:19.398906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:19.399013 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:19.399236 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:19.399240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:19.399349 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:58:19.399436 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:58:19.399554 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
I20260928 14:58:19.400086 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 7966
I20260928 14:58:19.400192 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:19.400417 2035 external_mini_cluster.cc:1428] Setting key d867cd6852d1c432b35b5b47dd622e4b
I20260928 14:58:19.400398 8095 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43295
I20260928 14:58:19.400513 8095 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:19.400709 8095 heartbeater.cc:511] Master 127.1.252.254:43295 requested a full tablet report, sending...
I20260928 14:58:19.401074 7636 ts_manager.cc:194] Registered new tserver with Master: b51769f43e914e37ba3432f5aa3cfb1f (127.1.252.195:38679)
I20260928 14:58:19.401535 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:58:19.401577 7636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:60343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/test-admin.keytab.
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.416072 2035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:58:19.417666 8102 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:19.416874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.1.252.193:37739)
0928 14:58:19.417078 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.417082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.417090 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:19.417114 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.417115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.417128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.417209 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.417580 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33826: BlockingRecv error: recv got EOF from 127.0.0.1:33826 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
W20260928 14:58:19.417765 2035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.418684 2035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.419540 2035 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:58:19.427513 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.420092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.1.252.193:37739)
0928 14:58:19.420153 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.420155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.420186 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:58:19.420265 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.420266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.420277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.420327 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.420757 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.421483 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.422264 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.422602 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.422686 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.424664 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:19.424672 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:19.424673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:19.424680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:19.426356 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.426711 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.426721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.426722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.426773 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.426984 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.426985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.426985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.427115 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:19.427283 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:19.427327 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 14:58:19.427400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 14:58:19.429186 8102 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:19.428628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.1.252.193:37739)
0928 14:58:19.428690 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.428693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.428736 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:58:19.428806 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.428807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.428817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.428871 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.429139 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33840: BlockingRecv error: recv got EOF from 127.0.0.1:33840 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:58:19.429250 2035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.430142 2035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.430951 2035 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:58:19.437148 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.431397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.1.252.193:37739)
0928 14:58:19.431440 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.431442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.431472 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:58:19.431519 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.431520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.431528 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.431569 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.431937 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.432392 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.433037 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.433219 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.433266 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.435146 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:19.435147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:19.435147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:19.435152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:19.436336 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.436623 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.436623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.436624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.436668 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.436805 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.436805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.436806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.436877 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:19.436953 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:19.437040 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 14:58:19.437095 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:58:19.438608 8102 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:19.438087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.1.252.193:37739)
0928 14:58:19.438143 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.438145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.438155 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:19.438226 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.438227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.438235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.438286 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.438565 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33868: BlockingRecv error: recv got EOF from 127.0.0.1:33868 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:58:19.438694 2035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607474232031-2035-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.439589 2035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:19.440456 2035 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 14:58:19.447325 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:19.440878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.1.252.193:37739)
0928 14:58:19.440957 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:19.440958 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:19.440966 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:19.441003 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:19.441004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:19.441011 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:19.441053 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:19.441513 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.442350 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.443020 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:19.443222 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:19.443305 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:19.445193 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:19.445195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:19.445195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:19.445202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:19.446281 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.446611 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.446612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.446613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.446670 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:19.446867 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:19.446868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:19.446868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:19.446967 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:19.447061 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:19.447172 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 14:58:19.447240 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:58:19.447835 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7698
I20260928 14:58:19.453081 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7832
I20260928 14:58:19.457528 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7966
I20260928 14:58:19.461880 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 7604
2026-09-28T14:58:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1860 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[8115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[8115](info): set up 2 sockets
Sep 28 14:58:19 dist-test-slave-sf70 krb5kdc[8115](info): commencing operation
krb5kdc: starting...
W20260928 14:58:20.489675 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:20 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab.
Entry for principal oryx/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab.
I20260928 14:58:20.606950 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:44901
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab
--principal=oryx/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:20.685925 8128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:20.686127 8128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:20.686151 8128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:20.687233 8128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:20.687268 8128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:20.687288 8128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:20.687300 8128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:20.687312 8128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:20.688833 8128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:44901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab
--principal=oryx/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:20.689028 8128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:20.689193 8128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:20.694118 8136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:20.694118 8134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:20.694134 8128 server_base.cc:1034] running on GCE node
W20260928 14:58:20.694150 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:20.694537 8128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:20.694829 8128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:20.695982 8128 hybrid_clock.cc:648] HybridClock initialized: now 1790607500695949 us; error 48 us; skew 500 ppm
Sep 28 14:58:20 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:20.704591 8128 init.cc:377] Logged in from keytab as oryx/127.1.252.254@KRBTEST.COM (short username oryx)
I20260928 14:58:20.705065 8128 webserver.cc:460] Webserver started at http://127.1.252.254:37859/ using document root <none> and password file <none>
I20260928 14:58:20.705320 8128 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:20.705387 8128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:20.705535 8128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:20.706585 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "ed6a19395db04a669b53520f581495ac"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "1ec18d0c8c752b9edf8b9baf934347bb"
server_key_iv: "92229be5695ec5c1ae325fe98a891e84"
server_key_version: "encryptionkey@0"
I20260928 14:58:20.706949 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "ed6a19395db04a669b53520f581495ac"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "1ec18d0c8c752b9edf8b9baf934347bb"
server_key_iv: "92229be5695ec5c1ae325fe98a891e84"
server_key_version: "encryptionkey@0"
I20260928 14:58:20.708350 8128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:20.709115 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:20.709283 8128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:20.709396 8128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "ed6a19395db04a669b53520f581495ac"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "1ec18d0c8c752b9edf8b9baf934347bb"
server_key_iv: "92229be5695ec5c1ae325fe98a891e84"
server_key_version: "encryptionkey@0"
I20260928 14:58:20.709472 8128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:20.717343 8128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:20.719496 8128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:20.719646 8128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:20.723618 8128 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:44901
I20260928 14:58:20.723649 8195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:44901 every 8 connection(s)
I20260928 14:58:20.724014 8128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:20.724934 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:20.727113 8196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Bootstrap starting.
I20260928 14:58:20.727674 8196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:20.728006 8196 log.cc:824] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:20.728663 8196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: No bootstrap required, opened a new log
I20260928 14:58:20.730228 8196 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } }
I20260928 14:58:20.730379 8196 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:20.730403 8196 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6a19395db04a669b53520f581495ac, State: Initialized, Role: FOLLOWER
I20260928 14:58:20.730571 8196 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } }
I20260928 14:58:20.730633 8196 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:20.730654 8196 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:20.730712 8196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:20.731293 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8128
I20260928 14:58:20.731421 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:20.731627 2035 external_mini_cluster.cc:1428] Setting key 34eba726a65f01b4f5a1b185b9696d91
I20260928 14:58:20.731742 8196 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } }
I20260928 14:58:20.731845 8196 leader_election.cc:302] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6a19395db04a669b53520f581495ac; no voters:
I20260928 14:58:20.732016 8196 leader_election.cc:288] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:20.732324 8201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:20.732411 8196 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:20.732501 8201 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 LEADER]: Becoming Leader. State: Replica: ed6a19395db04a669b53520f581495ac, State: Running, Role: LEADER
I20260928 14:58:20.732594 8201 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } }
I20260928 14:58:20.733026 8199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } }
I20260928 14:58:20.733134 8199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:20.733006 8201 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6a19395db04a669b53520f581495ac. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } }
I20260928 14:58:20.733208 8201 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:20.733419 8210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:20.733975 8210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:20.735532 8210 catalog_manager.cc:1347] Generated new cluster ID: e7dc26e1867b41bc9b01a635af6871f1
I20260928 14:58:20.735584 8210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:20 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:20.742182 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:20.732221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.1.252.254:44901)
0928 14:58:20.732487 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:20.732497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:20.732529 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:58:20.732649 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:20.732657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:20.732713 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:20.733710 (+ 997us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:20.733828 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:20.734947 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:20.735846 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:20.736293 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:20.736431 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:20.738675 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:20.738679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:20.738679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:20.738691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:20.740607 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:20.741006 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:20.741008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:20.741009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:20.741123 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:20.741333 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:20.741334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:20.741334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:20.741611 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:20.741748 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:20.741864 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:58:20.741988 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260928 14:58:20.754429 8210 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:20.754952 8210 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:20.757321 8210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Generated new TSK 0
I20260928 14:58:20.757478 8210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab.
Entry for principal oryx/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab.
I20260928 14:58:20.786933 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44901
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:20.876549 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:20.876756 8222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:20.876783 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:20.878005 8222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:20.878046 8222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:20.878075 8222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:20.879669 8222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:20.879926 8222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:20.880193 8222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:20.885872 8230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:20.885808 8227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:20.885821 8228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:20.885855 8222 server_base.cc:1034] running on GCE node
I20260928 14:58:20.886252 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:20.886473 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:20.887624 8222 hybrid_clock.cc:648] HybridClock initialized: now 1790607500887585 us; error 51 us; skew 500 ppm
Sep 28 14:58:20 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:20.890509 8222 init.cc:377] Logged in from keytab as oryx/127.1.252.193@KRBTEST.COM (short username oryx)
I20260928 14:58:20.890977 8222 webserver.cc:460] Webserver started at http://127.1.252.193:37083/ using document root <none> and password file <none>
I20260928 14:58:20.891175 8222 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:20.891218 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:20.891316 8222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:20.892757 8222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "f977979ca01c41698a9499eecdedc97d"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "cf0bdd505f7a5e1ee2d7a6cb426170ec"
server_key_iv: "b8c4230e04a81033e2298a5d22576214"
server_key_version: "encryptionkey@0"
I20260928 14:58:20.893277 8222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "f977979ca01c41698a9499eecdedc97d"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "cf0bdd505f7a5e1ee2d7a6cb426170ec"
server_key_iv: "b8c4230e04a81033e2298a5d22576214"
server_key_version: "encryptionkey@0"
I20260928 14:58:20.895360 8222 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:20.896509 8237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:20.896757 8222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:58:20.896864 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "f977979ca01c41698a9499eecdedc97d"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "cf0bdd505f7a5e1ee2d7a6cb426170ec"
server_key_iv: "b8c4230e04a81033e2298a5d22576214"
server_key_version: "encryptionkey@0"
I20260928 14:58:20.896950 8222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:20.909997 8222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:20.911795 8222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:20.911985 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:20.912225 8222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:20.912554 8222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:20.912590 8222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:20.912637 8222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:20.912665 8222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:20.918804 8222 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:40941
I20260928 14:58:20.918905 8350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:40941 every 8 connection(s)
I20260928 14:58:20.919232 8222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:20.921947 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8222
I20260928 14:58:20.922075 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:20.922289 2035 external_mini_cluster.cc:1428] Setting key e521f77a75507434c8fd8ce1684b5ac6
Sep 28 14:58:20 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:20.929831 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:20.920468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:42471 (local address 127.1.252.254:44901)
0928 14:58:20.920513 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:20.920515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:20.920524 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:20.920940 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:20.920941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:20.920954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:20.921040 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:20.922067 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:20.922762 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:20.923791 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:20.924095 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:20.924182 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:20.926975 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:20.926976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:20.926977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:20.926984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:20.928415 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:20.928836 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:20.928838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:20.928838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:20.928906 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:20.929176 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:20.929177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:20.929177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:20.929294 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:20.929370 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:20.929635 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:58:20.929722 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:58:20.930039 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:20.920106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:42471)
0928 14:58:20.920279 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:20.920317 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:20.920848 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:20.921118 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:20.921121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:20.921693 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:20.921959 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:20.921965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:20.922874 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:20.922874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:20.923673 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:20.923676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:20.924195 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:20.924196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:20.924353 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:20.924943 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:20.924955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:20.926854 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:20.928515 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:20.928522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:20.928524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:20.928743 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:20.928974 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:20.928974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:20.928975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:20.929104 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:20.929361 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:20.929367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:20.929505 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:58:20.929612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 14:58:20.929769 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260928 14:58:20.930518 8351 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44901
I20260928 14:58:20.930609 8351 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:20.930799 8351 heartbeater.cc:511] Master 127.1.252.254:44901 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 14:58:20.931394 8160 ts_manager.cc:194] Registered new tserver with Master: f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941)
I20260928 14:58:20.932181 8160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:42471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab.
Entry for principal oryx/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab.
I20260928 14:58:20.951568 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44901
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:21.034201 8356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:21.034400 8356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:21.034425 8356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:21.035490 8356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:21.035560 8356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:21.035606 8356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:21.037016 8356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:21.037192 8356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:21.037354 8356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.042446 8364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.042469 8361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.042433 8362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.042582 8356 server_base.cc:1034] running on GCE node
I20260928 14:58:21.042829 8356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.043032 8356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.044165 8356 hybrid_clock.cc:648] HybridClock initialized: now 1790607501044143 us; error 35 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.046651 8356 init.cc:377] Logged in from keytab as oryx/127.1.252.194@KRBTEST.COM (short username oryx)
I20260928 14:58:21.047078 8356 webserver.cc:460] Webserver started at http://127.1.252.194:35253/ using document root <none> and password file <none>
I20260928 14:58:21.047269 8356 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.047308 8356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.047402 8356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:21.048432 8356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "5a2d090deaf943aca44e10daf0327fa5"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "e54f8570be268fdb68d5ed58f7bcee79"
server_key_iv: "14783c4e9eff0a65e86ba415d84c320c"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.048727 8356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "5a2d090deaf943aca44e10daf0327fa5"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "e54f8570be268fdb68d5ed58f7bcee79"
server_key_iv: "14783c4e9eff0a65e86ba415d84c320c"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.049961 8356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:21.050639 8371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.050841 8356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:21.050936 8356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "5a2d090deaf943aca44e10daf0327fa5"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "e54f8570be268fdb68d5ed58f7bcee79"
server_key_iv: "14783c4e9eff0a65e86ba415d84c320c"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.051018 8356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.063830 8356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.065795 8356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:21.065960 8356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.066179 8356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:21.066478 8356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:21.066531 8356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.066571 8356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:21.066599 8356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.072348 8356 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:37937
I20260928 14:58:21.072391 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:37937 every 8 connection(s)
I20260928 14:58:21.072688 8356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:21.076218 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8356
I20260928 14:58:21.076328 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:21.076514 2035 external_mini_cluster.cc:1428] Setting key cf65af5a940ca5f142ffc772dd96c453
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:21.083184 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.073456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:36775)
0928 14:58:21.073669 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.073719 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.074198 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.074392 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.074396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.074940 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.075124 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.075128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.075866 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.075867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.076944 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.076947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.077372 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.077373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.077522 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.077981 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.077992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.079785 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.081665 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.081670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.081672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.081953 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.082174 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.082175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.082175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.082281 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.082583 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.082588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.082816 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.082909 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.083033 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260928 14:58:21.083180 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.073470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:36775 (local address 127.1.252.254:44901)
0928 14:58:21.073543 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.073545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.073816 (+ 271us) server_negotiation.cc:407] Connection header received
0928 14:58:21.074280 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.074281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.074291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.074341 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.075203 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.075775 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.077029 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.077292 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.077342 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.079902 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.079904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.079904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.079912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.081578 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.082036 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.082037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.082038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.082103 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.082347 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.082348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.082348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.082509 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.082586 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.082957 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.083045 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:58:21.083734 8485 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44901
I20260928 14:58:21.083844 8485 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:21.084026 8485 heartbeater.cc:511] Master 127.1.252.254:44901 requested a full tablet report, sending...
I20260928 14:58:21.084443 8160 ts_manager.cc:194] Registered new tserver with Master: 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:21.084853 8160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:36775
WARNING: no policy specified for oryx/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab.
Entry for principal oryx/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab.
I20260928 14:58:21.102794 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44901
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:21.181807 8490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:21.182010 8490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:21.182034 8490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:21.183176 8490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:21.183216 8490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:21.183243 8490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:21.184760 8490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:21.185000 8490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:21.185233 8490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.190209 8496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.190219 8498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.190219 8495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.190454 8490 server_base.cc:1034] running on GCE node
I20260928 14:58:21.190675 8490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.190912 8490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.192080 8490 hybrid_clock.cc:648] HybridClock initialized: now 1790607501192074 us; error 36 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.194427 8490 init.cc:377] Logged in from keytab as oryx/127.1.252.195@KRBTEST.COM (short username oryx)
I20260928 14:58:21.194849 8490 webserver.cc:460] Webserver started at http://127.1.252.195:33571/ using document root <none> and password file <none>
I20260928 14:58:21.195034 8490 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.195075 8490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.195168 8490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:21.196172 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "b45e90aacaa54746971e2d3d0706bb90"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5a09b92278bbdf93b7b5b82e814cf03b"
server_key_iv: "de29887fa110fcdf8216188e24c3bd43"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.196475 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "b45e90aacaa54746971e2d3d0706bb90"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5a09b92278bbdf93b7b5b82e814cf03b"
server_key_iv: "de29887fa110fcdf8216188e24c3bd43"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.197726 8490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:21.198393 8505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.198561 8490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:21.198624 8490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "b45e90aacaa54746971e2d3d0706bb90"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5a09b92278bbdf93b7b5b82e814cf03b"
server_key_iv: "de29887fa110fcdf8216188e24c3bd43"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.198673 8490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.219350 8490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.221398 8490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:21.221501 8490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.221925 8490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:21.222203 8490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:21.222237 8490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.222259 8490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:21.222272 8490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.228307 8490 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35627
I20260928 14:58:21.228361 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35627 every 8 connection(s)
I20260928 14:58:21.228706 8490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:21.237218 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8490
I20260928 14:58:21.237325 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:21.237540 2035 external_mini_cluster.cc:1428] Setting key 702393085291f5b99d9f9204ab66da11
I20260928 14:58:21.237959 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.229405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:44717 (local address 127.1.252.254:44901)
0928 14:58:21.229470 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.229472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.229652 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:58:21.230112 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.230113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.230130 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.230181 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.231044 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.231533 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.232645 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.232821 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.232920 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.235091 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.235092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.235092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.235101 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.236481 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.236889 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.236890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.236891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.236948 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.237203 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.237204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.237205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.237350 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.237424 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.237728 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.237864 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:58:21.238101 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.229374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:44717)
0928 14:58:21.229524 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.229566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.230029 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.230243 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.230246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.230784 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.230970 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.230974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.231621 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.231621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.232573 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.232575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.232926 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.232929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.233038 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.233599 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.233608 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.234989 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.236579 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.236586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.236588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.236803 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.237019 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.237020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.237020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.237134 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.237423 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.237431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.237565 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.237681 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.237844 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260928 14:58:21.238458 8619 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44901
I20260928 14:58:21.238572 8619 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:21.238749 8619 heartbeater.cc:511] Master 127.1.252.254:44901 requested a full tablet report, sending...
I20260928 14:58:21.239114 8160 ts_manager.cc:194] Registered new tserver with Master: b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627)
I20260928 14:58:21.239490 8160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:44717
I20260928 14:58:21.249277 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.252.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.253.keytab.
Entry for principal oryx/127.1.252.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.253.keytab.
I20260928 14:58:21.274920 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:35465
--webserver_interface=127.1.252.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.253.keytab
--principal=oryx/127.1.252.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:44901,127.1.252.253:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:21.355693 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:21.355926 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:21.355948 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:21.357108 8624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:21.357152 8624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:21.357167 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:21.357179 8624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:21.357192 8624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:21.358655 8624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:44901,127.1.252.253:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.253.keytab
--principal=oryx/127.1.252.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.252.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
W20260928 14:58:21.358759 8624 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.252.254:44901,127.1.252.253:35465'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:58:21.358942 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:21.359182 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.364187 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.364217 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.364287 8624 server_base.cc:1034] running on GCE node
W20260928 14:58:21.364202 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.364594 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.364825 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.365959 8624 hybrid_clock.cc:648] HybridClock initialized: now 1790607501365947 us; error 28 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.368417 8624 init.cc:377] Logged in from keytab as oryx/127.1.252.253@KRBTEST.COM (short username oryx)
I20260928 14:58:21.368821 8624 webserver.cc:460] Webserver started at http://127.1.252.253:46277/ using document root <none> and password file <none>
I20260928 14:58:21.369056 8624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.369122 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.369271 8624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:21.370294 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/instance:
uuid: "0140017d01bc48ccb7677bf277067392"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "f5b7e385bf4c7b2e597cc308cb855297"
server_key_iv: "94322be53edb2e74cf9cbb9403063635"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.370668 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal/instance:
uuid: "0140017d01bc48ccb7677bf277067392"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "f5b7e385bf4c7b2e597cc308cb855297"
server_key_iv: "94322be53edb2e74cf9cbb9403063635"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.371999 8624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:58:21.372735 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.372911 8624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:21.373008 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
uuid: "0140017d01bc48ccb7677bf277067392"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "f5b7e385bf4c7b2e597cc308cb855297"
server_key_iv: "94322be53edb2e74cf9cbb9403063635"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.373093 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.398237 8624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.400514 8624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:21.400673 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.404920 8624 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.253:35465
I20260928 14:58:21.404953 8691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.253:35465 every 8 connection(s)
I20260928 14:58:21.405288 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
I20260928 14:58:21.406236 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:21.409655 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8624
I20260928 14:58:21.410660 8692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.253@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
I20260928 14:58:21.421684 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.410926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.1.252.253:35465)
0928 14:58:21.411101 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.411110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.411141 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:58:21.411711 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.411719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.411788 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.412490 (+ 702us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.412687 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.414040 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.414772 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.415208 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.415308 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.418016 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.418025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.418025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.418032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.420227 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.420565 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.420568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.420568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.420663 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.420880 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.420881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.420882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.421089 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.421234 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.421348 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.421467 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260928 14:58:21.421716 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.411671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.1.252.254:44901)
0928 14:58:21.411732 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.411734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.411744 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:21.411908 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.411909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.411922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.411988 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.412948 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.413694 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.415160 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.415447 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.415526 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.418523 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.418524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.418524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.418532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.420133 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.420634 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.420635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.420635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.420688 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.420925 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.420927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.420927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.421150 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.421220 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.421486 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.421576 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:58:21.422077 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.411382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57346)
0928 14:58:21.411573 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.411617 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.411780 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.412072 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.412081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.412570 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.412852 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.412858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.413807 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.413809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.415069 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.415073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.415533 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.415534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.415605 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.416112 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.416121 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.418396 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.420222 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.420229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.420231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.420549 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.420761 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.420762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.420762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.420863 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.421250 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.421257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.421331 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.421800 (+ 469us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.421928 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260928 14:58:21.422637 8692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.253@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
I20260928 14:58:21.429919 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.422911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50330)
0928 14:58:21.422955 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.422965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.422999 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.423111 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.423112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.423281 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.423330 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.423332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.424226 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.424228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.425146 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.425149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.425499 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.425500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.425579 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.425941 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.425946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.427194 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.428952 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.428953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.428954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.429112 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.429334 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.429335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.429335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.429388 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.429623 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.429623 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.429658 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.429725 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.429817 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 14:58:21.429970 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.422972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.1.252.253:35465)
0928 14:58:21.423004 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.423006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.423036 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:58:21.423055 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.423055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.423067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.423103 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.423417 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.424154 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.425244 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.425426 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.425479 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.427303 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.427304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.427305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.427311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.428852 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.429212 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.429213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.429213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.429270 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.429459 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.429461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.429461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.429558 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.429630 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.429752 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.429863 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":16}
I20260928 14:58:21.431082 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Bootstrap starting.
I20260928 14:58:21.431701 8692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:21.432034 8692 log.cc:824] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:21.432798 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: No bootstrap required, opened a new log
I20260928 14:58:21.434262 8692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } }
I20260928 14:58:21.434374 8692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:21.434396 8692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0140017d01bc48ccb7677bf277067392, State: Initialized, Role: FOLLOWER
I20260928 14:58:21.434525 8692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } }
I20260928 14:58:21.434826 8692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:21.434811 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } } }
I20260928 14:58:21.434913 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:58:21.436700 8709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:58:21.436802 8709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:58:21.438524 8624 master_runner.cc:428] Detected that this master 0140017d01bc48ccb7677bf277067392 is joining an existing cluster
I20260928 14:58:21.438566 8624 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.252.253:35465
I20260928 14:58:21.447281 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.439517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57352)
0928 14:58:21.439685 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.439696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.439760 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.440136 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.440137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.440375 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.440509 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.440511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.441454 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.441455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.442566 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.442569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.443068 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.443069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.443155 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.443630 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.443635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.444116 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.445982 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.445983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.445984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.446237 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.446466 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.446467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.446468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.446536 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.446859 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.446860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.446911 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.446988 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.447103 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260928 14:58:21.447295 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.439548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.1.252.254:44901)
0928 14:58:21.439875 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.439878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.439994 (+ 116us) server_negotiation.cc:407] Connection header received
0928 14:58:21.440018 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.440019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.440031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.440088 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.440601 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.441367 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.442667 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.442969 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.443043 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.444217 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.444218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.444218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.444226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.445896 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.446326 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.446327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.446328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.446390 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.446611 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.446612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.446612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.446784 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.446858 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.447103 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.447186 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.254@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
I20260928 14:58:21.459676 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.449790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50346)
0928 14:58:21.449909 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.449950 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.450023 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.450174 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.450175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.450412 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.450454 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.450456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.451407 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.451408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.452073 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.452076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.452569 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.452570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.452627 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.453137 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.453142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.454684 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.458149 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.458158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.458160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.458470 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.458728 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.458729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.458730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.458799 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.459134 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.459143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.459209 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.459319 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.459498 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260928 14:58:21.460306 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.449843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.1.252.253:35465)
0928 14:58:21.449968 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.449970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.450004 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:58:21.450093 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.450094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.450107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.450172 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.450534 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.451304 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.452154 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.452487 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.452557 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.454777 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.454778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.454778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.454785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.458033 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.458565 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.458567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.458567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.458631 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.458886 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.458887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.458888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.459033 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.459100 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.460059 (+ 959us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.460157 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260928 14:58:21.461346 8160 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.252.253:35465: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0140017d01bc48ccb7677bf277067392"
member_type: NON_VOTER
last_known_addr {
host: "127.1.252.253"
port: 35465
}
attrs {
promote: true
}
}
dest_uuid: "ed6a19395db04a669b53520f581495ac"
cas_config_opid_index: -1
I20260928 14:58:21.461721 8160 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } }
I20260928 14:58:21.462353 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.439759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50338)
0928 14:58:21.439869 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.439880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.439940 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.440113 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.440114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.456074 (+ 15960us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.456126 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.456128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.457529 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.457529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.458364 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.458366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.458821 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.458821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.458887 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.459292 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.459296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.459679 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.461427 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.461427 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.461428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.461610 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.461768 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.461769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.461769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.461813 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.462039 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.462040 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.462084 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.462145 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.462252 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260928 14:58:21.462680 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.439965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.1.252.253:35465)
0928 14:58:21.440000 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.440001 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.440007 (+ 6us) server_negotiation.cc:407] Connection header received
0928 14:58:21.440023 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.440024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.440031 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.440106 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.456212 (+ 16106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.457446 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.458448 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.458726 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.458778 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.459764 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.459765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.459765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.459771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.461343 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.461673 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.461674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.461675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.461713 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.461866 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.461866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.461867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.461967 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.462020 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.462542 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.462606 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:21.464088 8720 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0140017d01bc48ccb7677bf277067392 (127.1.252.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } }
I20260928 14:58:21.464411 8666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:21.465086 8720 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.252.253:35465
I20260928 14:58:21.465166 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } }
I20260928 14:58:21.465219 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:21.465533 8666 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Refusing update from remote peer ed6a19395db04a669b53520f581495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 14:58:21.465932 8719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:21.465996 8624 master.cc:537] Master@127.1.252.253:35465 shutting down...
W20260928 14:58:21.468084 8145 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 14:58:21.468719 8145 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac -> Peer 0140017d01bc48ccb7677bf277067392 (127.1.252.253:35465): Couldn't send request to peer 0140017d01bc48ccb7677bf277067392. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:58:21.469913 8624 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:21.470011 8624 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:21.470063 8624 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: stopping tablet replica
I20260928 14:58:21.511148 8624 master.cc:552] Master@127.1.252.253:35465 shutdown complete.
I20260928 14:58:21.511240 8624 master_runner.cc:305] Clearing existing system tablet
I20260928 14:58:21.511381 8624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.511459 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.512462 8624 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.512953 8724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.513105 8624 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.513162 8624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.513203 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
uuid: "0140017d01bc48ccb7677bf277067392"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "f5b7e385bf4c7b2e597cc308cb855297"
server_key_iv: "94322be53edb2e74cf9cbb9403063635"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.513767 8624 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:21.515436 8624 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:58:21.515496 8624 log.cc:1196] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:58:21.515690 8624 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Deleting consensus metadata
I20260928 14:58:21.515820 8624 master_runner.cc:315] Copying system tablet from 127.1.252.254:44901
I20260928 14:58:21.516501 8624 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: tablet copy: Beginning tablet copy session from remote peer at address 127.1.252.254:44901
I20260928 14:58:21.522718 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.516828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57364)
0928 14:58:21.516952 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.516966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.517042 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.517181 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.517182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.517354 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.517400 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.517402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.518080 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.518081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.518619 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.518620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.519009 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.519010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.519075 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.519580 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.519584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.519865 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.521779 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.521780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.521780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.521943 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.522153 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.522153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.522153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.522203 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.522483 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.522483 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.522529 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.522579 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.522654 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260928 14:58:21.522902 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.516914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.1.252.254:44901)
0928 14:58:21.516966 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.516968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.517024 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:58:21.517108 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.517109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.517119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.517175 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.517487 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.517994 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.518708 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.518923 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.519009 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.520085 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.520087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.520087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.520096 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.521697 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.522019 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.522021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.522021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.522085 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.522268 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.522269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.522269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.522417 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.522491 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.522629 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.522711 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:58:21.523272 8180 tablet_copy_service.cc:140] P ed6a19395db04a669b53520f581495ac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0140017d01bc48ccb7677bf277067392 ({username='oryx', principal='oryx/127.1.252.253@KRBTEST.COM'} at 127.0.0.1:57364)
I20260928 14:58:21.523375 8180 tablet_copy_service.cc:161] P ed6a19395db04a669b53520f581495ac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0140017d01bc48ccb7677bf277067392 at {username='oryx', principal='oryx/127.1.252.253@KRBTEST.COM'} at 127.0.0.1:57364: session id = 0140017d01bc48ccb7677bf277067392-00000000000000000000000000000000
I20260928 14:58:21.524065 8180 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:58:21.524659 8624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:58:21.525550 8624 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: tablet copy: Starting download of 0 data blocks...
I20260928 14:58:21.525710 8624 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: tablet copy: Starting download of 1 WAL segments...
I20260928 14:58:21.526638 8624 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:58:21.527463 8180 tablet_copy_service.cc:342] P ed6a19395db04a669b53520f581495ac: Request end of tablet copy session 0140017d01bc48ccb7677bf277067392-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.252.253@KRBTEST.COM'} at 127.0.0.1:57364
I20260928 14:58:21.527522 8180 tablet_copy_service.cc:433] P ed6a19395db04a669b53520f581495ac: ending tablet copy session 0140017d01bc48ccb7677bf277067392-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0140017d01bc48ccb7677bf277067392
I20260928 14:58:21.529871 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.531036 8731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.531067 8732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.532999 8734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.533159 8624 server_base.cc:1034] running on GCE node
I20260928 14:58:21.533252 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.533385 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.534504 8624 hybrid_clock.cc:648] HybridClock initialized: now 1790607501534495 us; error 22 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.535953 8624 init.cc:377] Logged in from keytab as oryx/127.1.252.253@KRBTEST.COM (short username oryx)
I20260928 14:58:21.536214 8624 webserver.cc:460] Webserver started at http://127.1.252.253:45021/ using document root <none> and password file <none>
I20260928 14:58:21.536345 8624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.536382 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.537133 8624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:21.537706 8741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.537871 8624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.537928 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
uuid: "0140017d01bc48ccb7677bf277067392"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "f5b7e385bf4c7b2e597cc308cb855297"
server_key_iv: "94322be53edb2e74cf9cbb9403063635"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.537984 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.547102 8624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.548614 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.551555 8624 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.253:35465
I20260928 14:58:21.551576 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.253:35465 every 8 connection(s)
I20260928 14:58:21.551867 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
I20260928 14:58:21.552590 8794 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:58:21.552975 8794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Bootstrap starting.
I20260928 14:58:21.555222 8794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:58:21.555434 8794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: Bootstrap complete.
I20260928 14:58:21.555630 8794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } }
I20260928 14:58:21.555716 8794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0140017d01bc48ccb7677bf277067392, State: Initialized, Role: LEARNER
I20260928 14:58:21.555766 8794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } }
I20260928 14:58:21.555990 8794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:21.555976 8797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } }
I20260928 14:58:21.556070 8797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:58:21.557341 8808 catalog_manager.cc:1259] Loaded cluster ID: e7dc26e1867b41bc9b01a635af6871f1
I20260928 14:58:21.557408 8808 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: loading cluster ID for follower catalog manager: success
I20260928 14:58:21.558394 8808 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: acquiring CA information for follower catalog manager: success
I20260928 14:58:21.558663 8808 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:58:21.590485 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.584094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.1.252.253:35465)
0928 14:58:21.584236 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.584238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.584245 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:58:21.584266 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.584267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.584276 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.584346 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.584949 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.585595 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.586315 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.586527 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.586628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.587850 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.587851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.587852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.587859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.589330 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.589696 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.589697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.589698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.589758 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.589966 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.589967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.589967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.590078 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.590181 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.590281 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.590365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.1.252.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.252.keytab.
Entry for principal oryx/127.1.252.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.252.keytab.
I20260928 14:58:21.614924 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:38305
--webserver_interface=127.1.252.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.252.keytab
--principal=oryx/127.1.252.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:21.693946 8812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:21.694124 8812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:21.694145 8812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:21.695196 8812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:21.695231 8812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:21.695245 8812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:21.695257 8812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:21.695271 8812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:21.696722 8812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.252.keytab
--principal=oryx/127.1.252.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.252.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:21.696906 8812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:21.697067 8812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.701758 8820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.701921 8812 server_base.cc:1034] running on GCE node
W20260928 14:58:21.701771 8818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.701779 8817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.702224 8812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.702572 8812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.703732 8812 hybrid_clock.cc:648] HybridClock initialized: now 1790607501703718 us; error 24 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.706146 8812 init.cc:377] Logged in from keytab as oryx/127.1.252.252@KRBTEST.COM (short username oryx)
I20260928 14:58:21.706607 8812 webserver.cc:460] Webserver started at http://127.1.252.252:42453/ using document root <none> and password file <none>
I20260928 14:58:21.706840 8812 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.706915 8812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.707047 8812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:21.708148 8812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/instance:
uuid: "a2c7677e771a4fc0b39830a4304e5bea"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5f5e08b2c9f21c8b20b67976393deab8"
server_key_iv: "25d8ad7d115453484a189b8589028df6"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.708482 8812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal/instance:
uuid: "a2c7677e771a4fc0b39830a4304e5bea"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5f5e08b2c9f21c8b20b67976393deab8"
server_key_iv: "25d8ad7d115453484a189b8589028df6"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.709712 8812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:58:21.710429 8827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.710596 8812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.710690 8812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
uuid: "a2c7677e771a4fc0b39830a4304e5bea"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5f5e08b2c9f21c8b20b67976393deab8"
server_key_iv: "25d8ad7d115453484a189b8589028df6"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.710772 8812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.727687 8812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.729648 8812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:21.729805 8812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.734041 8812 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.252:38305
I20260928 14:58:21.734071 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.252:38305 every 8 connection(s)
I20260928 14:58:21.734473 8812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
I20260928 14:58:21.735410 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:21.736894 8880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:58:21.739176 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8812
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.252@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.252.252@KRBTEST.COM
I20260928 14:58:21.746279 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.737577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.1.252.254:44901)
0928 14:58:21.737650 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.737652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.737803 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:58:21.738288 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.738289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.738301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.738357 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.738984 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.739525 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.740814 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.741005 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.741055 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.743459 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.743460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.743461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.743467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.744978 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.745339 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.745340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.745340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.745390 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.745649 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.745650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.745650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.745759 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.745834 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.746111 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.746192 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:58:21.746455 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.737476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57366)
0928 14:58:21.737692 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.737725 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.738183 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.738464 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.738467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.738698 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.738881 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.738886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.739637 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.739638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.740731 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.740734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.741098 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.741099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.741231 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.741751 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.741759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.743350 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.745058 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.745065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.745068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.745277 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.745462 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.745463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.745464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.745571 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.745828 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.745834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.745967 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.746070 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.746216 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
I20260928 14:58:21.747093 8880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:58:21.748298 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.739621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.1.252.252:38305)
0928 14:58:21.739794 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.739798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.739815 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:58:21.739837 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.739838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.739857 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.739966 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.740495 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.741439 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.742204 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.742551 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.742607 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.745346 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.745349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.745349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.745354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.747020 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.747432 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.747434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.747435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.747492 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.747712 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.747713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.747713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.747848 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.748058 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.748123 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.748193 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.252@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
I20260928 14:58:21.753201 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.747330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50368)
0928 14:58:21.747374 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.747383 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.747424 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.747612 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.747613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.747743 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.747768 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.747769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.748743 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.748744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.749276 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.749278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.749675 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.749676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.749734 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.750056 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.750060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.751110 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.752434 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.752435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.752435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.752588 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.752753 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.752754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.752754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.752800 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.752993 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.752994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.753028 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.753078 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.753145 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 14:58:21.753221 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.747424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.1.252.253:35465)
0928 14:58:21.747490 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.747493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.747501 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:21.747527 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.747528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.747540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.747612 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.747844 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.748438 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.749363 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.749590 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.749675 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.751182 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.751183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.751183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.751190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.752361 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.752652 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.752652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.752653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.752699 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.752838 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.752838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.752839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.752939 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.752982 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.753101 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.753162 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:58:21.753662 8880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.252@KRBTEST.COM for oryx/127.1.252.252@KRBTEST.COM
I20260928 14:58:21.759855 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.753915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48902)
0928 14:58:21.753956 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.753966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.754010 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.754103 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.754103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.754203 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.754229 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.754230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.754857 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.754858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.755707 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.755713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.756196 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.756197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.756245 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.756635 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.756640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.757865 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.759108 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.759108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.759109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.759260 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.759437 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.759437 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.759437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.759485 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.759639 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.759639 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.759667 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.759711 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.759776 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 14:58:21.759974 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.753956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.1.252.252:38305)
0928 14:58:21.753987 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.753989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.753998 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:21.754053 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.754054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.754062 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.754101 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.754284 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.754778 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.755812 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.756114 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.756195 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.757937 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.757938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.757939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.757944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.759039 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.759322 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.759323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.759323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.759380 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.759523 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.759524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.759524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.759604 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.759645 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.759726 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.759816 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":15}
I20260928 14:58:21.761185 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Bootstrap starting.
I20260928 14:58:21.761997 8880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:21.762297 8880 log.cc:824] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:21.762988 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: No bootstrap required, opened a new log
I20260928 14:58:21.764483 8880 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } }
I20260928 14:58:21.764624 8880 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:21.764672 8880 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c7677e771a4fc0b39830a4304e5bea, State: Initialized, Role: FOLLOWER
I20260928 14:58:21.764802 8880 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } }
I20260928 14:58:21.765101 8880 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:21.765061 8885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } } }
I20260928 14:58:21.765162 8885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:58:21.766752 8896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:58:21.766814 8896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:58:21.768990 8812 master_runner.cc:428] Detected that this master a2c7677e771a4fc0b39830a4304e5bea is joining an existing cluster
I20260928 14:58:21.769026 8812 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.252.252:38305
I20260928 14:58:21.791417 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.770289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57374)
0928 14:58:21.770421 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.770431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.770488 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.770679 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.770680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.770846 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.770913 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.770914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.771844 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.771844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.772766 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.772768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.773294 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.773295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.773359 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.773794 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.773799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.788389 (+ 14590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.790259 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.790260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.790261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.790444 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.790679 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.790680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.790680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.790738 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.791036 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.791037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.791094 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.791160 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.791263 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 14:58:21.791704 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.770651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.1.252.253:35465)
0928 14:58:21.770693 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.770695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.771057 (+ 362us) server_negotiation.cc:407] Connection header received
0928 14:58:21.771120 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.771122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.771134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.771187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.771647 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.772423 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.773477 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.774325 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.774374 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.788690 (+ 14316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.788691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.788692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.788699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.790416 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.790902 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.790903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.790903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.790956 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.791164 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.791165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.791165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.791285 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.791328 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.791527 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.791615 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:21.791769 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.770313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.1.252.254:44901)
0928 14:58:21.770360 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.770362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.770476 (+ 114us) server_negotiation.cc:407] Connection header received
0928 14:58:21.770570 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.770571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.770583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.770614 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.770991 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.771747 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.772874 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.773168 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.773230 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.788494 (+ 15264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.788496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.788497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.788505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.790158 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.790549 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.790551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.790551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.790608 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.790807 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.790808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.790808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.790947 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.791011 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.791604 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.791677 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:21.792362 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.770328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50370)
0928 14:58:21.770994 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.771002 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.771063 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.771236 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.771237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.771493 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.771556 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.771558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.772503 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.772503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.773389 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.773390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.774472 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.774473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.774543 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.774991 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.774996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.788591 (+ 13595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.790496 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.790497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.790497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.790808 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.791032 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.791033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.791033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.791088 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.791375 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.791376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.791410 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.792147 (+ 737us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.792250 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":6,"thread_start_us":76,"threads_started":1}
I20260928 14:58:21.792891 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.770554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.1.252.252:38305)
0928 14:58:21.770586 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.770588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.771198 (+ 610us) server_negotiation.cc:407] Connection header received
0928 14:58:21.771287 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.771287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.771295 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.771321 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.771690 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.772413 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.773475 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.773732 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.773774 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.789001 (+ 15227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.789002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.789002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.789009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.791188 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.792134 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.792135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.792135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.792194 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.792415 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.792416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.792417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.792526 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.792565 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.792732 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.792805 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:58:21.793424 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.770513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48916)
0928 14:58:21.771153 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.771161 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.771226 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.771365 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.771366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.771566 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.771620 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.771622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.772503 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.772503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.773385 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.773387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.773836 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.773837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.773906 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.774292 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.774297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.788933 (+ 14636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.791272 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.791273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.791273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.791459 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.792268 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.792269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.792269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.792332 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.792602 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.792603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.792640 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.793197 (+ 557us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.793296 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":36,"thread_start_us":59,"threads_started":1}
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.254@KRBTEST.COM for oryx/127.1.252.252@KRBTEST.COM
I20260928 14:58:21.799685 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8222
I20260928 14:58:21.800805 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.794192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48924)
0928 14:58:21.794288 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.794299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.794347 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.794456 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.794457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.794618 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.794666 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.794667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.795230 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.795231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.796079 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.796082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.796392 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.796393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.796461 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.796787 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.796792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.798049 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.799654 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.799655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.799655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.799878 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.800089 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.800090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.800090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.800148 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.800428 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.800429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.800473 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.800537 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.800644 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 14:58:21.801189 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.794244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.1.252.252:38305)
0928 14:58:21.794287 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.794289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.794332 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:58:21.794403 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.794403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.794410 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.794451 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.794730 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.795157 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.796152 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.796321 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.796374 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.798134 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.798135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.798135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.798142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.799565 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.799976 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.799977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.799977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.800028 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.800205 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.800206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.800206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.800331 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.800400 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.800982 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.801060 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:58:21.801648 8160 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.252.252:38305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea"
member_type: NON_VOTER
last_known_addr {
host: "127.1.252.252"
port: 38305
}
attrs {
promote: true
}
}
dest_uuid: "ed6a19395db04a669b53520f581495ac"
cas_config_opid_index: 5
I20260928 14:58:21.801903 8160 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } }
I20260928 14:58:21.802433 8905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.803529 8905 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.252.252:38305
I20260928 14:58:21.803761 8812 master.cc:537] Master@127.1.252.252:38305 shutting down...
I20260928 14:58:21.803748 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.803819 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:21.806488 8854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:21.807225 8854 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Refusing update from remote peer ed6a19395db04a669b53520f581495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:58:21.807497 8720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:21.808198 8812 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:21.808282 8812 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:58:21.808297 8145 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac -> Peer a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252:38305): Couldn't send request to peer a2c7677e771a4fc0b39830a4304e5bea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:58:21.808324 8812 tablet_replica.cc:330] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: stopping tablet replica
I20260928 14:58:21.809736 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:40941
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:44901
--webserver_port=37083
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:58:21.811571 8812 master.cc:552] Master@127.1.252.252:38305 shutdown complete.
I20260928 14:58:21.811614 8812 master_runner.cc:305] Clearing existing system tablet
I20260928 14:58:21.811748 8812 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.811790 8812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.812665 8812 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.813015 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.802585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50382)
0928 14:58:21.802811 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.802822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.802860 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.809425 (+ 6565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.809426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.809428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:21.809464 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.809466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.810394 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.810395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.811817 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.811820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.812766 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.812767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.812833 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.812866 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.812916 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.812949 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260928 14:58:21.813253 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.802623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.1.252.253:35465)
0928 14:58:21.809266 (+ 6643us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.809268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.809276 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:21.809297 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.809298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.809317 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.809406 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:21.809546 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.810304 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.811949 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.812602 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.812668 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.812734 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.813150 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.813189 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6621}
I20260928 14:58:21.813318 8908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.813472 8764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 LEARNER]: Refusing update from remote peer ed6a19395db04a669b53520f581495ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:58:21.813621 8812 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.813674 8812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:58:21.813649 8720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 14:58:21.813700 8812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
uuid: "a2c7677e771a4fc0b39830a4304e5bea"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5f5e08b2c9f21c8b20b67976393deab8"
server_key_iv: "25d8ad7d115453484a189b8589028df6"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.814136 8764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.814615 8812 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:21.815369 8797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6a19395db04a669b53520f581495ac. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.815433 8797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:58:21.815459 8797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: NON_VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: true } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.815485 8797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:58:21.816206 8812 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:58:21.816270 8812 log.cc:1196] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:58:21.816354 8720 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: attempting to promote NON_VOTER 0140017d01bc48ccb7677bf277067392 to VOTER
I20260928 14:58:21.816493 8812 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Deleting consensus metadata
I20260928 14:58:21.816533 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } }
I20260928 14:58:21.816718 8812 master_runner.cc:315] Copying system tablet from 127.1.252.254:44901
I20260928 14:58:21.817106 8764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 LEARNER]: Refusing update from remote peer ed6a19395db04a669b53520f581495ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:58:21.817149 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.816933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48930)
0928 14:58:21.817066 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.817115 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 14:58:21.817289 8719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 14:58:21.817324 8812 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: tablet copy: Beginning tablet copy session from remote peer at address 127.1.252.254:44901
I20260928 14:58:21.817664 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.817534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48932)
0928 14:58:21.817608 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.817633 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 14:58:21.818156 8145 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac -> Peer a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252:38305): Couldn't send request to peer a2c7677e771a4fc0b39830a4304e5bea. Status: Network error: Client connection negotiation failed: client connection to 127.1.252.252:38305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:58:21.818332 8905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0140017d01bc48ccb7677bf277067392 (127.1.252.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.818553 8764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0140017d01bc48ccb7677bf277067392 (127.1.252.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.819144 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.819213 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:21.819258 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.819296 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:21.819537 8910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.819630 8910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:58:21.825824 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.817619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.1.252.254:44901)
0928 14:58:21.817653 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.817655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.817924 (+ 269us) server_negotiation.cc:407] Connection header received
0928 14:58:21.818015 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.818016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.818025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.818053 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.819654 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.820440 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.821537 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.821845 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.821919 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.822994 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.822995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.822995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.823002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.824640 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.825039 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.825040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.825041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.825100 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.825304 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.825305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.825305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.825437 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.825500 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.825656 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.825728 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:58:21.825834 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.817746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57382)
0928 14:58:21.817868 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.817876 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.817948 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.818108 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.818108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.819498 (+ 1390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.819556 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.819559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.820564 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.820565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.821440 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.821443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.821930 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.821931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.821997 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.822511 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.822516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.822902 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.824728 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.824729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.824729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.824963 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.825171 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.825171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.825172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.825238 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.825503 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.825504 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.825551 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.825623 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.825727 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260928 14:58:21.826444 8180 tablet_copy_service.cc:140] P ed6a19395db04a669b53520f581495ac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a2c7677e771a4fc0b39830a4304e5bea ({username='oryx', principal='oryx/127.1.252.252@KRBTEST.COM'} at 127.0.0.1:57382)
I20260928 14:58:21.826495 8180 tablet_copy_service.cc:161] P ed6a19395db04a669b53520f581495ac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a2c7677e771a4fc0b39830a4304e5bea at {username='oryx', principal='oryx/127.1.252.252@KRBTEST.COM'} at 127.0.0.1:57382: session id = a2c7677e771a4fc0b39830a4304e5bea-00000000000000000000000000000000
I20260928 14:58:21.827028 8180 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:58:21.827561 8812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:58:21.828907 8812 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: tablet copy: Starting download of 0 data blocks...
I20260928 14:58:21.829052 8812 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: tablet copy: Starting download of 1 WAL segments...
I20260928 14:58:21.829795 8812 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:58:21.830654 8180 tablet_copy_service.cc:342] P ed6a19395db04a669b53520f581495ac: Request end of tablet copy session a2c7677e771a4fc0b39830a4304e5bea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.252.252@KRBTEST.COM'} at 127.0.0.1:57382
I20260928 14:58:21.830708 8180 tablet_copy_service.cc:433] P ed6a19395db04a669b53520f581495ac: ending tablet copy session a2c7677e771a4fc0b39830a4304e5bea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a2c7677e771a4fc0b39830a4304e5bea
I20260928 14:58:21.833427 8812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.834517 8917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.836764 8918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.836911 8920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.844086 8812 server_base.cc:1034] running on GCE node
I20260928 14:58:21.844197 8812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.844338 8812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.845474 8812 hybrid_clock.cc:648] HybridClock initialized: now 1790607501845450 us; error 43 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.847024 8812 init.cc:377] Logged in from keytab as oryx/127.1.252.252@KRBTEST.COM (short username oryx)
I20260928 14:58:21.847280 8812 webserver.cc:460] Webserver started at http://127.1.252.252:40233/ using document root <none> and password file <none>
I20260928 14:58:21.847400 8812 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.847437 8812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.848209 8812 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:21.848798 8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.848915 8812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:21.848953 8812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
uuid: "a2c7677e771a4fc0b39830a4304e5bea"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5f5e08b2c9f21c8b20b67976393deab8"
server_key_iv: "25d8ad7d115453484a189b8589028df6"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.849011 8812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.856319 8812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.858292 8812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.861244 8812 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.252:38305
I20260928 14:58:21.861466 8979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.252:38305 every 8 connection(s)
I20260928 14:58:21.861599 8812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
I20260928 14:58:21.862437 8980 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:58:21.862968 8980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Bootstrap starting.
I20260928 14:58:21.865804 8980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:58:21.866052 8980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Bootstrap complete.
I20260928 14:58:21.866240 8980 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } }
I20260928 14:58:21.866324 8980 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2c7677e771a4fc0b39830a4304e5bea, State: Initialized, Role: LEARNER
I20260928 14:58:21.866355 8980 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } }
I20260928 14:58:21.866537 8980 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:21.866547 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:21.866773 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: This master's current role is: LEARNER
I20260928 14:58:21.868291 8994 catalog_manager.cc:1259] Loaded cluster ID: e7dc26e1867b41bc9b01a635af6871f1
I20260928 14:58:21.868376 8994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: loading cluster ID for follower catalog manager: success
I20260928 14:58:21.869524 8994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: acquiring CA information for follower catalog manager: success
I20260928 14:58:21.869808 8994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 14:58:21.906342 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:21.906569 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:21.906607 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:21.907763 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:21.907829 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:21.907873 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:21.909379 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:40941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=37083
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:21.909646 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:21.909868 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:21.915691 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.915762 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:21.915787 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:21.916078 8906 server_base.cc:1034] running on GCE node
I20260928 14:58:21.916288 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:21.916515 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:21.917677 8906 hybrid_clock.cc:648] HybridClock initialized: now 1790607501917663 us; error 28 us; skew 500 ppm
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:21.920027 8906 init.cc:377] Logged in from keytab as oryx/127.1.252.193@KRBTEST.COM (short username oryx)
I20260928 14:58:21.920481 8906 webserver.cc:460] Webserver started at http://127.1.252.193:37083/ using document root <none> and password file <none>
I20260928 14:58:21.920708 8906 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:21.920787 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:21.922115 8906 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:21.922686 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.922865 8906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:21.922974 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "f977979ca01c41698a9499eecdedc97d"
format_stamp: "Formatted at 2026-09-28 14:58:20 on dist-test-slave-sf70"
server_key: "cf0bdd505f7a5e1ee2d7a6cb426170ec"
server_key_iv: "b8c4230e04a81033e2298a5d22576214"
server_key_version: "encryptionkey@0"
I20260928 14:58:21.923297 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:21.939774 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:21.941537 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:21.941697 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:21.941927 8906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:21.942219 8906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:21.942272 8906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.942312 8906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:21.942339 8906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:21.948220 8906 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:40941
I20260928 14:58:21.948268 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:40941 every 8 connection(s)
I20260928 14:58:21.948587 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:21.955547 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 8906
I20260928 14:58:21.955706 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8356
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.252@KRBTEST.COM
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
Sep 28 14:58:21 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607501, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:21.960974 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.949633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:47459 (local address 127.1.252.253:35465)
0928 14:58:21.949680 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.949682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.949845 (+ 163us) server_negotiation.cc:407] Connection header received
0928 14:58:21.950675 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.950676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.950689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.950741 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.951635 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.952417 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953649 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.953945 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953999 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.957842 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.957844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.957844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.957852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.959650 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960096 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.960097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.960098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.960153 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960339 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.960340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.960341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.960486 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.960528 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.960730 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.960817 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:21.961020 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.949300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:33207 (local address 127.1.252.252:38305)
0928 14:58:21.949730 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.949732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.949741 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:21.950728 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.950730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.950742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.950816 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.951636 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.952718 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953586 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.953898 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953960 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.956834 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.956835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.956836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.956843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.959197 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.959724 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.959725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.959726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.959791 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960108 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.960109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.960110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.960255 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.960315 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.960798 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.960904 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":59,"threads_started":1}
I20260928 14:58:21.961565 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.949571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.1.252.193:47459)
0928 14:58:21.949782 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.949792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.950583 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.950822 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.950826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.951335 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.951524 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.951530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.952494 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.952495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.953575 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.953576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.954043 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.954044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.954103 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.954771 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.954781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.957349 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.959742 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.959743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.959744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.960013 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.960221 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.960222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.960277 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.960575 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.960576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.960619 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.961064 (+ 445us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.961276 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":12,"thread_start_us":56,"threads_started":1}
I20260928 14:58:21.961567 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.949264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.1.252.193:33207)
0928 14:58:21.949502 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.949511 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.950570 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.950822 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.950826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.951318 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.951524 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.951530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.952803 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.952804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.953487 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.953490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.954020 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.954021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.954103 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.954770 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.954781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.956735 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.959302 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.959310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.959312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.959635 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.959862 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.959863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.959864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.960025 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.960342 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.960348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.960509 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.960621 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.961348 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":46,"thread_start_us":97,"threads_started":1}
I20260928 14:58:21.962220 9125 heartbeater.cc:347] Connected to a master server at 127.1.252.253:35465
I20260928 14:58:21.962247 9123 heartbeater.cc:347] Connected to a master server at 127.1.252.252:38305
I20260928 14:58:21.962319 9125 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:21.962319 9123 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:21.962538 9125 heartbeater.cc:511] Master 127.1.252.253:35465 requested a full tablet report, sending...
I20260928 14:58:21.962541 9123 heartbeater.cc:511] Master 127.1.252.252:38305 requested a full tablet report, sending...
I20260928 14:58:21.962598 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.949324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:44983 (local address 127.1.252.254:44901)
0928 14:58:21.949383 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:21.949385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:21.949574 (+ 189us) server_negotiation.cc:407] Connection header received
0928 14:58:21.950695 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:21.950697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:21.950710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:21.950742 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:21.951620 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.952189 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953683 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.953874 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953959 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.958424 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:21.958425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:21.958426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:21.958435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:21.960316 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960703 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.960704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.960705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.960765 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960967 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:21.960968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:21.960968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:21.961095 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:21.961142 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:21.962421 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0928 14:58:21.962506 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:58:21.963097 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:21.949253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:44983)
0928 14:58:21.949452 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:21.949495 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:21.950571 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:21.950822 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:21.950826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:21.951328 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:21.951523 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.951528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.952292 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.952293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.953573 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:21.953576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:21.953968 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:21.953969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:21.954088 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:21.954830 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:21.954834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:21.958301 (+ 3467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:21.960405 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.960406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.960621 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.960841 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:21.960841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:21.960842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:21.960898 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:21.961544 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:21.961544 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:21.961587 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:58:21.962910 (+ 1323us) client_negotiation.cc:240] Negotiation successful
0928 14:58:21.963009 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":55,"thread_start_us":108,"threads_started":1}
I20260928 14:58:21.963090 8758 ts_manager.cc:194] Registered new tserver with Master: f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941)
I20260928 14:58:21.963389 8944 ts_manager.cc:194] Registered new tserver with Master: f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941)
I20260928 14:58:21.963706 9124 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44901
I20260928 14:58:21.963759 9124 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:21.963802 9124 heartbeater.cc:511] Master 127.1.252.254:44901 requested a full tablet report, sending...
I20260928 14:58:21.964066 8160 ts_manager.cc:194] Re-registered known tserver with Master: f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941)
I20260928 14:58:21.964524 8160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:44983
I20260928 14:58:21.964591 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:37937
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:44901
--webserver_port=35253
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:22.046561 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:22.046785 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:22.046824 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:22.048032 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:22.048091 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:22.048135 9131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:22.049666 9131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:37937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=35253
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:22.049922 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:22.050151 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:22.055150 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:22.055231 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:22.055156 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:22.055447 9131 server_base.cc:1034] running on GCE node
I20260928 14:58:22.055689 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:22.056043 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:22.057253 9131 hybrid_clock.cc:648] HybridClock initialized: now 1790607502057237 us; error 36 us; skew 500 ppm
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:22.060146 9131 init.cc:377] Logged in from keytab as oryx/127.1.252.194@KRBTEST.COM (short username oryx)
I20260928 14:58:22.060601 9131 webserver.cc:460] Webserver started at http://127.1.252.194:35253/ using document root <none> and password file <none>
I20260928 14:58:22.060819 9131 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:22.060900 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:22.062217 9131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:22.062842 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:22.063015 9131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:22.063115 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "5a2d090deaf943aca44e10daf0327fa5"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "e54f8570be268fdb68d5ed58f7bcee79"
server_key_iv: "14783c4e9eff0a65e86ba415d84c320c"
server_key_version: "encryptionkey@0"
I20260928 14:58:22.063445 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:22.071099 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:22.072852 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:22.073014 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:22.073225 9131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:22.073524 9131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:22.073576 9131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:22.073611 9131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:22.073637 9131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:22.079614 9131 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:37937
I20260928 14:58:22.079651 9259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:37937 every 8 connection(s)
I20260928 14:58:22.080075 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.252@KRBTEST.COM
I20260928 14:58:22.090061 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 9131
I20260928 14:58:22.090240 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8490
I20260928 14:58:22.090977 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.080825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:54937 (local address 127.1.252.254:44901)
0928 14:58:22.080881 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.080883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.081119 (+ 236us) server_negotiation.cc:407] Connection header received
0928 14:58:22.082206 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.082207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.082220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.082305 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.083161 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.083745 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.084784 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.084997 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085077 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.087414 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.087415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.087415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.087423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.089352 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.089794 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.089796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.089796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.089857 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090202 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.090203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.090203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.090387 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.090432 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.090802 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.090882 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:58:22.090976 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.080781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:55199 (local address 127.1.252.253:35465)
0928 14:58:22.080883 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.080886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.081297 (+ 411us) server_negotiation.cc:407] Connection header received
0928 14:58:22.082206 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.082207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.082220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.082286 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.083176 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.083957 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085082 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.085292 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085371 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.087882 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.087902 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.087902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.087910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.089792 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090159 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.090160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.090161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.090218 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090436 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.090437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.090438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.090562 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.090603 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.090802 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.090882 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:58:22.091462 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.081188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:47045 (local address 127.1.252.252:38305)
0928 14:58:22.081234 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.081236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.082325 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 14:58:22.082440 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.082441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.082456 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.082515 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.083165 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.083960 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085061 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.085364 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.088452 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.088454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.088454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.088462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.090138 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090578 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.090579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.090580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.090633 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090833 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.090834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.090834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.090969 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.091012 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.091213 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.091308 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:58:22.091535 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.080786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.1.252.194:55199)
0928 14:58:22.081235 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.081245 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.082103 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.082369 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.082373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.082827 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.083051 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.083057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.084058 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.084059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.084978 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.084979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.085369 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.085435 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.085927 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.085933 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.087756 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.089874 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.089875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.089875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.090075 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.090293 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.090294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.090347 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.090648 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.090649 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.090681 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.091112 (+ 431us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.091291 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":53,"thread_start_us":125,"threads_started":1}
I20260928 14:58:22.092221 9262 heartbeater.cc:347] Connected to a master server at 127.1.252.253:35465
I20260928 14:58:22.092314 9262 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:22.092480 9262 heartbeater.cc:511] Master 127.1.252.253:35465 requested a full tablet report, sending...
I20260928 14:58:22.092948 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.080786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:54937)
0928 14:58:22.080983 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.081024 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.082093 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.082373 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.082375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.082808 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.083051 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.083055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.083835 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.083836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.084690 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.084692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.085139 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.085285 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.085764 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.085773 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.087284 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.089451 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.089457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.089459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.089695 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.089941 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.089941 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.089942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.090066 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.090480 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.090487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.090637 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.091436 (+ 799us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.092179 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":41,"thread_start_us":123,"threads_started":1}
I20260928 14:58:22.092948 8758 ts_manager.cc:194] Registered new tserver with Master: 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:22.093299 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.081106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.1.252.194:47045)
0928 14:58:22.082270 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.082280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.082350 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.082538 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.082539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.082827 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.083051 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.083057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.084058 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.084059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.084974 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.084977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.085504 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.085505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.085564 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.086104 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.086109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.088343 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.090231 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.090232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.090495 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.090698 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.090699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.090699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.090762 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.091057 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.091058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.091101 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.091579 (+ 478us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.093084 (+ 1505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":1445,"thread_start_us":57,"threads_started":1}
I20260928 14:58:22.093631 9260 heartbeater.cc:347] Connected to a master server at 127.1.252.252:38305
I20260928 14:58:22.093667 9261 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44901
I20260928 14:58:22.093681 9260 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:22.093742 9261 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:22.093789 9260 heartbeater.cc:511] Master 127.1.252.252:38305 requested a full tablet report, sending...
I20260928 14:58:22.093822 9261 heartbeater.cc:511] Master 127.1.252.254:44901 requested a full tablet report, sending...
I20260928 14:58:22.094056 8160 ts_manager.cc:194] Re-registered known tserver with Master: 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:22.094064 8944 ts_manager.cc:194] Registered new tserver with Master: 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:22.094448 8160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:54937
I20260928 14:58:22.096524 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:35627
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:44901
--webserver_port=33571
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:22.177340 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:22.177564 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:22.177597 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:22.178757 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:22.178824 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:22.178869 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:22.180490 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:35627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=33571
--tserver_master_addrs=127.1.252.254:44901,127.1.252.253:35465,127.1.252.252:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:22.180740 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:22.180986 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:22.186036 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:22.186038 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:22.186048 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:22.186267 9267 server_base.cc:1034] running on GCE node
I20260928 14:58:22.186493 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:22.186767 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:22.187928 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790607502187909 us; error 34 us; skew 500 ppm
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:22.190377 9267 init.cc:377] Logged in from keytab as oryx/127.1.252.195@KRBTEST.COM (short username oryx)
I20260928 14:58:22.190788 9267 webserver.cc:460] Webserver started at http://127.1.252.195:33571/ using document root <none> and password file <none>
I20260928 14:58:22.191007 9267 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:22.191088 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:22.192564 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:22.193136 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:22.193311 9267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:22.193423 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "b45e90aacaa54746971e2d3d0706bb90"
format_stamp: "Formatted at 2026-09-28 14:58:21 on dist-test-slave-sf70"
server_key: "5a09b92278bbdf93b7b5b82e814cf03b"
server_key_iv: "de29887fa110fcdf8216188e24c3bd43"
server_key_version: "encryptionkey@0"
I20260928 14:58:22.193742 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:22.210114 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:22.212422 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:22.212577 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:22.212802 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:22.213145 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:22.213196 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:22.213235 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:22.213263 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:22.218918 9267 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35627
I20260928 14:58:22.218956 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35627 every 8 connection(s)
I20260928 14:58:22.219302 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:22.220894 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 9267
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.253@KRBTEST.COM
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.252@KRBTEST.COM
I20260928 14:58:22.231859 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.222068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.1.252.254:44901)
0928 14:58:22.222197 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.222198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.222679 (+ 481us) server_negotiation.cc:407] Connection header received
0928 14:58:22.222763 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.222764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.222778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.222848 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.223691 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.224526 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.225536 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.225821 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.225861 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.227813 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.227814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.227815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.227822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.230517 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.230926 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.230927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.230927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.230990 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.231203 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.231204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.231205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.231338 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.231431 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.231655 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.231732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 14:58:22.232707 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.220324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.1.252.195:38773)
0928 14:58:22.220513 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.220523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.220904 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.221246 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.221247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.221702 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.221991 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.221997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.223014 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.223014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.224394 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.224397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.224917 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.224918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.225057 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.225801 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.225811 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.228498 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.230731 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.230738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.230740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.231076 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.231263 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.231264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.231265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.231411 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.231722 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.231728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.232167 (+ 439us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.232285 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.232460 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":60,"thread_start_us":55,"threads_started":1}
I20260928 14:58:22.233932 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.220499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:38773 (local address 127.1.252.253:35465)
0928 14:58:22.220569 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.220571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.220580 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:22.221072 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.221073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.221086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.221136 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.222101 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.222899 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.224490 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.224791 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.224847 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.228585 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.228587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.228587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.228595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.230320 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.231152 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.231153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.231154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.231206 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.231494 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.231495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.231495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.231640 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.233704 (+ 2064us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.233754 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.233824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:58:22.234259 9398 heartbeater.cc:347] Connected to a master server at 127.1.252.253:35465
I20260928 14:58:22.234342 9398 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:22.234521 9398 heartbeater.cc:511] Master 127.1.252.253:35465 requested a full tablet report, sending...
I20260928 14:58:22.234849 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.220285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:49431 (local address 127.1.252.252:38305)
0928 14:58:22.220334 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.220336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.220351 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:58:22.221010 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.221012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.221026 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.221061 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.222104 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.222890 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.224497 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.224801 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.224841 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.229383 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.229384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.229384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.229391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.232662 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.234056 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.234057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.234057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.234111 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.234343 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.234344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.234344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.234460 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.234500 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.234686 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.234764 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":23}
I20260928 14:58:22.234894 8758 ts_manager.cc:194] Registered new tserver with Master: b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627)
I20260928 14:58:22.235150 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.220043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.1.252.195:49431)
0928 14:58:22.220233 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.220280 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.220880 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.221701 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.221702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.221839 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.221991 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.221997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.223013 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.223014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.224398 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.224400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.224919 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.224920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.225057 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.227200 (+ 2143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.227205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.229285 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.233777 (+ 4492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.233778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.233778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.233972 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.234179 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.234180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.234180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.234272 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.234546 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.234546 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.234584 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.234963 (+ 379us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.235061 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":46,"spinlock_wait_cycles":6528,"thread_start_us":95,"threads_started":1}
I20260928 14:58:22.235435 9396 heartbeater.cc:347] Connected to a master server at 127.1.252.252:38305
I20260928 14:58:22.235474 9396 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:22.235530 9396 heartbeater.cc:511] Master 127.1.252.252:38305 requested a full tablet report, sending...
I20260928 14:58:22.233434 8160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:58:22.235776 8944 ts_manager.cc:194] Registered new tserver with Master: b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627)
W20260928 14:58:22.235963 8160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:22.236480 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.222431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.1.252.253:35465)
0928 14:58:22.222533 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.222535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.222847 (+ 312us) server_negotiation.cc:407] Connection header received
0928 14:58:22.223203 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.223204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.223216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.223254 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.225323 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.226078 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.227457 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.227791 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.227828 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.230289 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.230290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.230290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.230296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.233638 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.235537 (+ 1899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.235539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.235539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.235595 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.235807 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.235808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.235808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.235970 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.236048 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.236272 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.236362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260928 14:58:22.237133 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.222594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.1.252.252:38305)
0928 14:58:22.222690 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.222691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.223043 (+ 352us) server_negotiation.cc:407] Connection header received
0928 14:58:22.223066 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.223067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.223079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.223134 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.224636 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.225480 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.226560 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.226894 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.226957 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.228789 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.228790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.228790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.228797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.233414 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.233762 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.233763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.233764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.233825 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.236001 (+ 2176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.236003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.236003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.236170 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.236254 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.236948 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.237043 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607502, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:22.240592 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.220043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:55849)
0928 14:58:22.220233 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.220280 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.220880 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.221208 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.221213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.223272 (+ 2059us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.228654 (+ 5382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.228656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.229570 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.229571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.230425 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.230427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.230977 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.230978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.231053 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.231996 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.232001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.237490 (+ 5489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.239386 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.239387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.239387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.239541 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.239752 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.239752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.239753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.239820 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.240230 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.240231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.240277 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.240343 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.240464 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260928 14:58:22.241323 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.220016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:55849 (local address 127.1.252.254:44901)
0928 14:58:22.220067 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.220069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.220364 (+ 295us) server_negotiation.cc:407] Connection header received
0928 14:58:22.221038 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.221039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.221052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.221084 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.228742 (+ 7658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.229472 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.230524 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.230838 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.230893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.237599 (+ 6706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.237601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.237601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.237608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.239308 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.239603 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.239604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.239604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.239682 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.240011 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.240012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.240012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.240156 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.240237 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.241163 (+ 926us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.241236 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:58:22.241595 9397 heartbeater.cc:347] Connected to a master server at 127.1.252.254:44901
I20260928 14:58:22.241663 9397 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:22.241724 9397 heartbeater.cc:511] Master 127.1.252.254:44901 requested a full tablet report, sending...
I20260928 14:58:22.241966 8160 ts_manager.cc:194] Re-registered known tserver with Master: b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627)
I20260928 14:58:22.242482 8160 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:55849
I20260928 14:58:22.245064 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.240533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.1.252.194:37937)
0928 14:58:22.240933 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.240937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.241144 (+ 207us) server_negotiation.cc:407] Connection header received
0928 14:58:22.241226 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.241228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.241244 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.241340 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:22.241612 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.242536 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.243779 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.244566 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.244610 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.244687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.244924 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.244967 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":92,"threads_started":1}
I20260928 14:58:22.245314 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.240434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37937 (local address 127.0.0.1:59796)
0928 14:58:22.241079 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.241089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.241149 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.241334 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.241335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.241337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:22.241388 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.241390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.242628 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.242629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.243686 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.243689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.244751 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.244751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.244817 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.244821 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.245178 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.245232 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":87,"threads_started":1}
Sep 28 14:58:22 dist-test-slave-sf70 krb5kdc[8115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607500, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.254@KRBTEST.COM for oryx/127.1.252.195@KRBTEST.COM
I20260928 14:58:22.246129 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.240848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40941 (local address 127.0.0.1:44318)
0928 14:58:22.242585 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.242595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.242645 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.242900 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.242901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.242903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:22.242947 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.242948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.244088 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.244089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.245005 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.245008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.245899 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.245900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.245958 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.245963 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.246012 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.246051 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":73,"threads_started":1}
I20260928 14:58:22.246059 9194 tablet_service.cc:1467] Processing CreateTablet for tablet cb474242a68b4fe1a57b81ddec34a7a2 (DEFAULT_TABLE table=test-table [id=7f4c9c3982f74a84b32e9240a163c170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:22.246433 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb474242a68b4fe1a57b81ddec34a7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:22.246433 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.240933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.1.252.193:40941)
0928 14:58:22.241095 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.241099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.242650 (+ 1551us) server_negotiation.cc:407] Connection header received
0928 14:58:22.242721 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.242723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.242756 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.242831 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:22.243027 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.244005 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.245126 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.245733 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.245800 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.245863 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.246288 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.246330 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 14:58:22.247020 9057 tablet_service.cc:1467] Processing CreateTablet for tablet cb474242a68b4fe1a57b81ddec34a7a2 (DEFAULT_TABLE table=test-table [id=7f4c9c3982f74a84b32e9240a163c170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:22.247347 9057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb474242a68b4fe1a57b81ddec34a7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:22.250008 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.240348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35627 (local address 127.0.0.1:37786)
0928 14:58:22.240461 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.240472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.240511 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.240935 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.240936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.241095 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:22.241136 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.241138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.242546 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.242547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.243319 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.243321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.244090 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.244090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.244199 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.244635 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:22.244640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:22.246065 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:22.248899 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.248900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.248901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.249066 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.249273 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:22.249274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:22.249274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:22.249330 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:22.249694 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:22.249695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:22.249739 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.249817 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.249920 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":84}
I20260928 14:58:22.250023 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.240584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.1.252.195:35627)
0928 14:58:22.240753 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.240758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.240768 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:22.240803 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.240805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.240819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.241212 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.241261 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.242460 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.243407 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.243840 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.244114 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.246152 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.246160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.246160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.246167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.248800 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.249154 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.249157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.249157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.249219 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.249403 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.249404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.249405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.249556 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.249708 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.249843 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.249929 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260928 14:58:22.250617 9330 tablet_service.cc:1467] Processing CreateTablet for tablet cb474242a68b4fe1a57b81ddec34a7a2 (DEFAULT_TABLE table=test-table [id=7f4c9c3982f74a84b32e9240a163c170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:22.250823 9423 tablet_bootstrap.cc:492] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Bootstrap starting.
I20260928 14:58:22.250916 9330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb474242a68b4fe1a57b81ddec34a7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:22.251430 9423 tablet_bootstrap.cc:654] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:22.251758 9423 log.cc:824] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:22.252060 9422 tablet_bootstrap.cc:492] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Bootstrap starting.
I20260928 14:58:22.252674 9422 tablet_bootstrap.cc:654] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:22.252691 9423 tablet_bootstrap.cc:492] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: No bootstrap required, opened a new log
I20260928 14:58:22.252748 9423 ts_tablet_manager.cc:1397] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:22.253000 9422 log.cc:824] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:22.253777 9422 tablet_bootstrap.cc:492] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: No bootstrap required, opened a new log
I20260928 14:58:22.253844 9422 ts_tablet_manager.cc:1397] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:22.254094 9426 tablet_bootstrap.cc:492] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Bootstrap starting.
I20260928 14:58:22.254649 9426 tablet_bootstrap.cc:654] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:22.254738 9423 raft_consensus.cc:348] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.254912 9423 raft_consensus.cc:374] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:22.254958 9423 raft_consensus.cc:729] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f977979ca01c41698a9499eecdedc97d, State: Initialized, Role: FOLLOWER
I20260928 14:58:22.254982 9426 log.cc:824] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:22.255072 9423 consensus_queue.cc:260] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.255350 9423 ts_tablet_manager.cc:1428] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:58:22.255716 9124 heartbeater.cc:503] Master 127.1.252.254:44901 was elected leader, sending a full tablet report...
I20260928 14:58:22.255781 9426 tablet_bootstrap.cc:492] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: No bootstrap required, opened a new log
I20260928 14:58:22.255841 9426 ts_tablet_manager.cc:1397] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:22.256472 9422 raft_consensus.cc:348] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.256606 9422 raft_consensus.cc:374] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:22.256692 9422 raft_consensus.cc:729] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2d090deaf943aca44e10daf0327fa5, State: Initialized, Role: FOLLOWER
I20260928 14:58:22.256814 9422 consensus_queue.cc:260] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.257089 9422 ts_tablet_manager.cc:1428] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 14:58:22.257169 9261 heartbeater.cc:503] Master 127.1.252.254:44901 was elected leader, sending a full tablet report...
I20260928 14:58:22.258447 9426 raft_consensus.cc:348] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.258558 9426 raft_consensus.cc:374] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:22.258579 9426 raft_consensus.cc:729] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e90aacaa54746971e2d3d0706bb90, State: Initialized, Role: FOLLOWER
I20260928 14:58:22.258677 9426 consensus_queue.cc:260] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.258870 9426 ts_tablet_manager.cc:1428] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:58:22.258980 9397 heartbeater.cc:503] Master 127.1.252.254:44901 was elected leader, sending a full tablet report...
I20260928 14:58:22.276010 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.273645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48950)
0928 14:58:22.273707 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.273720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.273777 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.273917 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.273918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.273920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:22.273963 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.273965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.274554 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.274554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.275241 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.275243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.275771 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.275772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.275818 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.275821 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.275867 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.275954 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:58:22.276022 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.273714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.1.252.252:38305)
0928 14:58:22.273771 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.273773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.273781 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:22.273831 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.273831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.273849 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.273899 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:22.274016 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.274496 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.275315 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.275699 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.275756 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.275801 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.275930 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.275973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:58:22.276362 8954 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 14:58:22.277191 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6a19395db04a669b53520f581495ac. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:22.277288 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: This master's current role is: LEARNER
W20260928 14:58:22.330825 9263 tablet.cc:2369] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:58:22.451287 9126 tablet.cc:2369] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:58:22.470870 9399 tablet.cc:2369] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:22.475780 9428 raft_consensus.cc:482] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:22.475961 9428 raft_consensus.cc:504] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.476331 9428 leader_election.cc:288] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627), 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:22.480263 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.476594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:41217 (local address 127.1.252.194:37937)
0928 14:58:22.476644 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.476646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.476724 (+ 78us) server_negotiation.cc:407] Connection header received
0928 14:58:22.476834 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.476835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.476845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.476923 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:22.477106 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.477857 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.479347 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.479875 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.479977 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.479989 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.480153 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.480198 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:22.480274 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.476603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:48235 (local address 127.1.252.195:35627)
0928 14:58:22.476644 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.476646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.476814 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:58:22.476848 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.476849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.476874 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.476966 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:22.477103 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.477855 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.479343 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.479993 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.480059 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.480129 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.480185 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.480221 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:58:22.480283 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.476521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35627 (local address 127.1.252.193:48235)
0928 14:58:22.476745 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.476755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.476805 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.476976 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.476977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.476978 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:22.477023 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.477026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.477952 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.477953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.479251 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.479254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.480062 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.480063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.480119 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.480122 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.480178 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.480208 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":49,"thread_start_us":57,"threads_started":1}
I20260928 14:58:22.480289 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.476521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37937 (local address 127.1.252.193:41217)
0928 14:58:22.476639 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:22.476651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:22.476739 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:22.476932 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:22.476933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:22.476935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:22.477023 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.477025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.477953 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.477954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.479251 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:22.479254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.479978 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.479979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:22.480071 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.480076 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:22.480135 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:58:22.480191 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":65,"thread_start_us":59,"threads_started":1}
I20260928 14:58:22.480676 9350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb474242a68b4fe1a57b81ddec34a7a2" candidate_uuid: "f977979ca01c41698a9499eecdedc97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e90aacaa54746971e2d3d0706bb90" is_pre_election: true
I20260928 14:58:22.480825 9350 raft_consensus.cc:2454] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f977979ca01c41698a9499eecdedc97d in term 0.
I20260928 14:58:22.480885 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb474242a68b4fe1a57b81ddec34a7a2" candidate_uuid: "f977979ca01c41698a9499eecdedc97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2d090deaf943aca44e10daf0327fa5" is_pre_election: true
I20260928 14:58:22.481006 9214 raft_consensus.cc:2454] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f977979ca01c41698a9499eecdedc97d in term 0.
I20260928 14:58:22.481097 9013 leader_election.cc:302] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45e90aacaa54746971e2d3d0706bb90, f977979ca01c41698a9499eecdedc97d; no voters:
I20260928 14:58:22.481269 9428 raft_consensus.cc:2781] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:22.481312 9428 raft_consensus.cc:482] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:22.481326 9428 raft_consensus.cc:3037] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:22.481987 9428 raft_consensus.cc:504] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.482131 9428 leader_election.cc:288] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [CANDIDATE]: Term 1 election: Requested vote from peers b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627), 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:22.482308 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb474242a68b4fe1a57b81ddec34a7a2" candidate_uuid: "f977979ca01c41698a9499eecdedc97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2d090deaf943aca44e10daf0327fa5"
I20260928 14:58:22.482395 9214 raft_consensus.cc:3037] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:22.483325 9214 raft_consensus.cc:2454] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f977979ca01c41698a9499eecdedc97d in term 1.
I20260928 14:58:22.483469 9350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb474242a68b4fe1a57b81ddec34a7a2" candidate_uuid: "f977979ca01c41698a9499eecdedc97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e90aacaa54746971e2d3d0706bb90"
I20260928 14:58:22.483525 9010 leader_election.cc:302] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2d090deaf943aca44e10daf0327fa5, f977979ca01c41698a9499eecdedc97d; no voters:
I20260928 14:58:22.483552 9350 raft_consensus.cc:3037] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:22.483614 9428 raft_consensus.cc:2781] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:22.483745 9428 raft_consensus.cc:686] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 1 LEADER]: Becoming Leader. State: Replica: f977979ca01c41698a9499eecdedc97d, State: Running, Role: LEADER
I20260928 14:58:22.483843 9428 consensus_queue.cc:237] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } }
I20260928 14:58:22.484414 9350 raft_consensus.cc:2454] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f977979ca01c41698a9499eecdedc97d in term 1.
I20260928 14:58:22.484898 8160 catalog_manager.cc:5495] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d reported cstate change: term changed from 0 to 1, leader changed from <none> to f977979ca01c41698a9499eecdedc97d (127.1.252.193). New cstate: current_term: 1 leader_uuid: "f977979ca01c41698a9499eecdedc97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } health_report { overall_health: HEALTHY } } }
I20260928 14:58:22.485471 9438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: NON_VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: true } } }
I20260928 14:58:22.485581 9438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:22.486214 8720 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: attempting to promote NON_VOTER a2c7677e771a4fc0b39830a4304e5bea to VOTER
I20260928 14:58:22.486519 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } }
I20260928 14:58:22.486975 8764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Refusing update from remote peer ed6a19395db04a669b53520f581495ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 14:58:22.486969 8954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 LEARNER]: Refusing update from remote peer ed6a19395db04a669b53520f581495ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 14:58:22.487178 9438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 14:58:22.487272 8719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 14:58:22.487910 8719 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.488168 8764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.488715 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.488762 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.488811 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:22.488819 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:22.488927 8954 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.489039 9441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.489116 9441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:58:22.489826 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.488731 9438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed6a19395db04a669b53520f581495ac" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:22.489921 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:58:22.489923 9438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:22.528012 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.525542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.1.252.193:40941)
0928 14:58:22.525606 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.525608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.525617 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:22.525678 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.525679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.525701 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.525791 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:22.525936 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.526424 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.527199 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.527415 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.527492 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.527659 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:22.527764 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:22.527860 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.527907 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.527930 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:58:22.529830 9350 raft_consensus.cc:1261] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Refusing update from remote peer f977979ca01c41698a9499eecdedc97d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:22.529830 9214 raft_consensus.cc:1261] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Refusing update from remote peer f977979ca01c41698a9499eecdedc97d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:22.530202 9436 consensus_queue.cc:1035] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:22.530342 9428 consensus_queue.cc:1035] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:22.534421 9445 mvcc.cc:205] Tried to move back new op lower bound from 7334328330359435264 to 7334328330174558208. Current Snapshot: MvccSnapshot[applied={T|T < 7334328330359435264}]
I20260928 14:58:22.534482 9444 mvcc.cc:205] Tried to move back new op lower bound from 7334328330359435264 to 7334328330174558208. Current Snapshot: MvccSnapshot[applied={T|T < 7334328330359435264 or (T in {7334328330359435264})}]
I20260928 14:58:22.536859 8160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:22.536959 8160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:22.538367 8160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed6a19395db04a669b53520f581495ac: Sending DeleteTablet for 3 replicas of tablet cb474242a68b4fe1a57b81ddec34a7a2
I20260928 14:58:22.538724 9057 tablet_service.cc:1514] Processing DeleteTablet for tablet cb474242a68b4fe1a57b81ddec34a7a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:22 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:44318
I20260928 14:58:22.538763 9330 tablet_service.cc:1514] Processing DeleteTablet for tablet cb474242a68b4fe1a57b81ddec34a7a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:22 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:37786
I20260928 14:58:22.538813 9194 tablet_service.cc:1514] Processing DeleteTablet for tablet cb474242a68b4fe1a57b81ddec34a7a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:22 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59796
I20260928 14:58:22.539062 9453 tablet_replica.cc:330] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: stopping tablet replica
I20260928 14:58:22.539085 9452 tablet_replica.cc:330] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: stopping tablet replica
I20260928 14:58:22.539186 9452 raft_consensus.cc:2229] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:22.539193 9453 raft_consensus.cc:2229] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:22.539302 9452 raft_consensus.cc:2258] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:22.539323 9453 raft_consensus.cc:2258] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:22.539827 9453 ts_tablet_manager.cc:1905] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:22.540163 9452 ts_tablet_manager.cc:1905] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:22.540899 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8128
I20260928 14:58:22.542004 9453 ts_tablet_manager.cc:1918] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:22.542131 9453 log.cc:1196] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/cb474242a68b4fe1a57b81ddec34a7a2
I20260928 14:58:22.542479 9453 ts_tablet_manager.cc:1939] T cb474242a68b4fe1a57b81ddec34a7a2 P b45e90aacaa54746971e2d3d0706bb90: Deleting consensus metadata
I20260928 14:58:22.543280 9452 ts_tablet_manager.cc:1918] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:22.543371 9452 log.cc:1196] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/cb474242a68b4fe1a57b81ddec34a7a2
I20260928 14:58:22.543602 9452 ts_tablet_manager.cc:1939] T cb474242a68b4fe1a57b81ddec34a7a2 P f977979ca01c41698a9499eecdedc97d: Deleting consensus metadata
I20260928 14:58:22.544011 9454 tablet_replica.cc:330] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: stopping tablet replica
I20260928 14:58:22.544114 9454 raft_consensus.cc:2229] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:22.544209 9454 raft_consensus.cc:2258] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:22.544620 9454 ts_tablet_manager.cc:1905] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:22.546542 9454 ts_tablet_manager.cc:1918] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:22.546613 9454 log.cc:1196] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/cb474242a68b4fe1a57b81ddec34a7a2
I20260928 14:58:22.546866 9454 ts_tablet_manager.cc:1939] T cb474242a68b4fe1a57b81ddec34a7a2 P 5a2d090deaf943aca44e10daf0327fa5: Deleting consensus metadata
W20260928 14:58:22.547366 9010 connection.cc:535] server connection from 127.0.0.1:44318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:58:22.547478 9150 connection.cc:535] server connection from 127.0.0.1:59796 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:58:22.547667 9283 connection.cc:535] server connection from 127.0.0.1:37786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:58:22.548570 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.539874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.1.252.252:38305)
0928 14:58:22.539929 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.539931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.539938 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:58:22.539956 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.539957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.539966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.540011 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.540797 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.542074 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.543063 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.543406 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.543477 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.545429 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.545430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.545431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.545438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.547128 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.547667 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.547668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.547668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.547728 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.547969 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.547970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.547970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.548101 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.548176 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.548397 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.548479 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:58:22.556552 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.549292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.1.252.253:35465)
0928 14:58:22.549334 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.549336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.549541 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:58:22.549657 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.549658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.549671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.549731 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.550206 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.550990 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.551783 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.552121 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.552190 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.553599 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.553600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.553601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.553608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.555104 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.555691 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.555692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.555692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.555755 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.556017 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.556018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.556018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.556153 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.556249 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.556338 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.556465 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:22.562616 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:22.549465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.1.252.252:38305)
0928 14:58:22.549502 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:22.549504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:22.556538 (+ 7034us) server_negotiation.cc:407] Connection header received
0928 14:58:22.557012 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:22.557013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:22.557021 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:22.557087 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:22.557571 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.558024 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.558689 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:22.558962 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:22.559030 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:22.560216 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:22.560217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:22.560218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:22.560224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:22.561544 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.561883 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.561884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.561884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.561936 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:22.562132 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:22.562133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:22.562133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:22.562240 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:22.562337 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:22.562455 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:58:22.562522 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:23.534965 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:23.534404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:60365)
0928 14:58:23.534847 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:23.534909 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":66,"threads_started":1}
I20260928 14:58:23.534965 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:23.534413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:34205)
0928 14:58:23.534807 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:23.534924 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":62,"threads_started":1}
I20260928 14:58:23.535175 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:23.534623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:55595)
0928 14:58:23.535053 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:23.535128 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":45,"threads_started":1}
W20260928 14:58:23.535249 9397 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:44901 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
W20260928 14:58:23.535403 9124 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:44901 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
W20260928 14:58:23.535487 9261 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:44901 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
I20260928 14:58:24.038991 9465 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed6a19395db04a669b53520f581495ac)
I20260928 14:58:24.039213 9465 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } }
I20260928 14:58:24.039366 9466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed6a19395db04a669b53520f581495ac)
I20260928 14:58:24.039557 9465 leader_election.cc:288] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed6a19395db04a669b53520f581495ac (127.1.252.254:44901), 0140017d01bc48ccb7677bf277067392 (127.1.252.253:35465)
I20260928 14:58:24.039501 9466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } }
I20260928 14:58:24.039969 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.039636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57940)
0928 14:58:24.039861 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.039929 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":46,"threads_started":1}
W20260928 14:58:24.040203 8928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
I20260928 14:58:24.040313 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.040062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57942)
0928 14:58:24.040200 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.040270 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
W20260928 14:58:24.040431 8742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
I20260928 14:58:24.040680 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.040562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57948)
0928 14:58:24.040619 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.040652 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 14:58:24.040880 8928 leader_election.cc:334] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed6a19395db04a669b53520f581495ac (127.1.252.254:44901): Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
I20260928 14:58:24.041018 9466 leader_election.cc:288] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed6a19395db04a669b53520f581495ac (127.1.252.254:44901), a2c7677e771a4fc0b39830a4304e5bea (127.1.252.252:38305)
I20260928 14:58:24.041173 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.040913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57960)
0928 14:58:24.041059 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.041142 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260928 14:58:24.041330 8742 leader_election.cc:334] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed6a19395db04a669b53520f581495ac (127.1.252.254:44901): Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
I20260928 14:58:24.043038 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.039676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:35465 (local address 127.0.0.1:50404)
0928 14:58:24.040029 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.040041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.040107 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.040321 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.040323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.040326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.040435 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.040438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.041485 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.041486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.042233 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.042235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.042795 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.042796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.042859 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.042863 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.042930 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.042971 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260928 14:58:24.043035 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.039738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.1.252.253:35465)
0928 14:58:24.040134 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.040136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.040146 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:24.040169 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.040170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.040183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.040543 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.040607 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.041378 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.042322 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.042713 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.042796 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.042806 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.042936 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.042973 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":50,"threads_started":1}
I20260928 14:58:24.043313 8764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2c7677e771a4fc0b39830a4304e5bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0140017d01bc48ccb7677bf277067392" is_pre_election: true
I20260928 14:58:24.043427 8764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c7677e771a4fc0b39830a4304e5bea in term 1.
I20260928 14:58:24.043586 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.040688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:38305 (local address 127.0.0.1:48972)
0928 14:58:24.040740 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.040751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.040797 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.041219 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.041220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.041223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.041268 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.041270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.041952 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.041953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.042706 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.042709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.043349 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.043350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.043419 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.043424 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.043484 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.043526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:58:24.043605 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.040966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.1.252.252:38305)
0928 14:58:24.041085 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.041087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.041095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:24.041115 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.041116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.041132 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.041216 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.041352 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.041857 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.042801 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.043276 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.043344 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.043354 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.043507 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.043549 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260928 14:58:24.043721 8929 leader_election.cc:302] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0140017d01bc48ccb7677bf277067392, a2c7677e771a4fc0b39830a4304e5bea; no voters: ed6a19395db04a669b53520f581495ac
I20260928 14:58:24.043828 9465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:58:24.043877 9465 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed6a19395db04a669b53520f581495ac)
I20260928 14:58:24.043922 9465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:58:24.043833 8954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0140017d01bc48ccb7677bf277067392" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2c7677e771a4fc0b39830a4304e5bea" is_pre_election: true
I20260928 14:58:24.044620 9465 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } }
I20260928 14:58:24.044749 8954 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0140017d01bc48ccb7677bf277067392 in current term 2: Already voted for candidate a2c7677e771a4fc0b39830a4304e5bea in this term.
I20260928 14:58:24.044765 9465 leader_election.cc:288] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [CANDIDATE]: Term 2 election: Requested vote from peers ed6a19395db04a669b53520f581495ac (127.1.252.254:44901), 0140017d01bc48ccb7677bf277067392 (127.1.252.253:35465)
I20260928 14:58:24.044948 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.044833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57968)
0928 14:58:24.044873 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.044924 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 14:58:24.044930 8764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2c7677e771a4fc0b39830a4304e5bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0140017d01bc48ccb7677bf277067392"
I20260928 14:58:24.044991 8764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:58:24.044968 8743 leader_election.cc:302] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0140017d01bc48ccb7677bf277067392; no voters: a2c7677e771a4fc0b39830a4304e5bea, ed6a19395db04a669b53520f581495ac
I20260928 14:58:24.045318 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.045209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57978)
0928 14:58:24.045269 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.045297 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:58:24.045432 8928 leader_election.cc:334] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed6a19395db04a669b53520f581495ac (127.1.252.254:44901): Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
I20260928 14:58:24.045830 8764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c7677e771a4fc0b39830a4304e5bea in term 2.
I20260928 14:58:24.045951 9466 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:58:24.046007 8929 leader_election.cc:302] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0140017d01bc48ccb7677bf277067392, a2c7677e771a4fc0b39830a4304e5bea; no voters: ed6a19395db04a669b53520f581495ac
I20260928 14:58:24.046079 9465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:58:24.046219 9465 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [term 2 LEADER]: Becoming Leader. State: Replica: a2c7677e771a4fc0b39830a4304e5bea, State: Running, Role: LEADER
I20260928 14:58:24.046308 9465 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } }
I20260928 14:58:24.046684 9475 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2c7677e771a4fc0b39830a4304e5bea. Latest consensus state: current_term: 2 leader_uuid: "a2c7677e771a4fc0b39830a4304e5bea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:24.046789 9475 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:24.046972 9477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:24.047349 9477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:24.047432 9477 catalog_manager.cc:1259] Loaded cluster ID: e7dc26e1867b41bc9b01a635af6871f1
I20260928 14:58:24.047469 9477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:24.047662 9477 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:24.047748 9477 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Loaded TSK: 0
I20260928 14:58:24.047950 9477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:24.099193 8943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:24.101121 8943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:24.102118 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.101994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:57996)
0928 14:58:24.102053 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.102092 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:58:24.102145 8764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [term 2 FOLLOWER]: Refusing update from remote peer a2c7677e771a4fc0b39830a4304e5bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 14:58:24.102463 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.102351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.0.0.1:58008)
0928 14:58:24.102399 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.102437 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:58:24.102524 9475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 14:58:24.102639 8928 consensus_peers.cc:487] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea -> Peer ed6a19395db04a669b53520f581495ac (127.1.252.254:44901): Couldn't send request to peer ed6a19395db04a669b53520f581495ac. Status: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:58:24.103093 9466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2c7677e771a4fc0b39830a4304e5bea. Latest consensus state: current_term: 2 leader_uuid: "a2c7677e771a4fc0b39830a4304e5bea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:24.103196 9466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:58:24.103618 9475 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a2c7677e771a4fc0b39830a4304e5bea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:24.103719 9475 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:24.103787 9479 mvcc.cc:205] Tried to move back new op lower bound from 7334328336800468992 to 7334328336574418944. Current Snapshot: MvccSnapshot[applied={T|T < 7334328336800468992}]
I20260928 14:58:24.103924 9466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a2c7677e771a4fc0b39830a4304e5bea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ed6a19395db04a669b53520f581495ac" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44901 } } peers { permanent_uuid: "0140017d01bc48ccb7677bf277067392" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 35465 } attrs { promote: false } } peers { permanent_uuid: "a2c7677e771a4fc0b39830a4304e5bea" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 38305 } attrs { promote: false } } }
I20260928 14:58:24.104010 9466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0140017d01bc48ccb7677bf277067392 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:58:24.108629 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.105402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40941 (local address 127.0.0.1:44344)
0928 14:58:24.105442 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.105451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.105490 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.105801 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.105802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.105804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.105857 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.105858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.106616 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.106617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.107624 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.107626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.108407 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.108408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.108467 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.108472 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.108528 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.108567 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 14:58:24.108819 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.105454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.1.252.193:40941)
0928 14:58:24.105665 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.105668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.105678 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:24.105706 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.105707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.105719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.105785 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.105940 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.106518 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.107717 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.108328 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.108400 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.108412 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.108547 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.108602 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":60,"threads_started":1}
I20260928 14:58:24.108839 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.105462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35627 (local address 127.0.0.1:37796)
0928 14:58:24.105550 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.105560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.105608 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.106111 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.106112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.106114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.106141 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.106142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.107031 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.107032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.107821 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.107823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.108615 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.108616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.108680 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.108685 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.108731 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.108775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 14:58:24.109025 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.105472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37937 (local address 127.0.0.1:59798)
0928 14:58:24.105585 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.105593 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.105639 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.105989 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.105990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.105992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.106024 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.106025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.106968 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.106969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.108067 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.108069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.108769 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.108771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.108831 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.108835 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.108858 (+ 23us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.108922 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":28,"thread_start_us":77,"threads_started":1}
I20260928 14:58:24.109018 9057 tablet_service.cc:1467] Processing CreateTablet for tablet c3d4b5b8f678405a9425874c64ba05e6 (DEFAULT_TABLE table=test-table [id=cdd30670b41443f99b01f4ede5ecf06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:24.109135 9057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d4b5b8f678405a9425874c64ba05e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:24.109164 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.105696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.1.252.195:35627)
0928 14:58:24.105979 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.105981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.105989 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:24.106014 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.106015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.106025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.106210 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.106246 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.106940 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.107938 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.108522 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.108998 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.109012 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.109031 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.109074 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":67,"threads_started":1}
I20260928 14:58:24.109331 9330 tablet_service.cc:1467] Processing CreateTablet for tablet c3d4b5b8f678405a9425874c64ba05e6 (DEFAULT_TABLE table=test-table [id=cdd30670b41443f99b01f4ede5ecf06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:24.109339 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.105662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.1.252.194:37937)
0928 14:58:24.105832 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.105834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.105843 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:24.105886 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.105887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.105899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.106089 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.106127 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.106886 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.108153 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.108673 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.109179 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.109198 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.109216 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.109266 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260928 14:58:24.109432 9330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d4b5b8f678405a9425874c64ba05e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:24.110608 9487 tablet_bootstrap.cc:492] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d: Bootstrap starting.
I20260928 14:58:24.110826 9488 tablet_bootstrap.cc:492] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Bootstrap starting.
I20260928 14:58:24.110952 9194 tablet_service.cc:1467] Processing CreateTablet for tablet c3d4b5b8f678405a9425874c64ba05e6 (DEFAULT_TABLE table=test-table [id=cdd30670b41443f99b01f4ede5ecf06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:24.111043 9487 tablet_bootstrap.cc:654] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:24.111102 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d4b5b8f678405a9425874c64ba05e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:24.111220 9488 tablet_bootstrap.cc:654] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:24.111613 9487 tablet_bootstrap.cc:492] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d: No bootstrap required, opened a new log
I20260928 14:58:24.111657 9487 ts_tablet_manager.cc:1397] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:24.111738 9487 raft_consensus.cc:348] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.111788 9487 raft_consensus.cc:374] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:24.111802 9487 raft_consensus.cc:729] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f977979ca01c41698a9499eecdedc97d, State: Initialized, Role: FOLLOWER
I20260928 14:58:24.111858 9487 consensus_queue.cc:260] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.112056 9487 ts_tablet_manager.cc:1428] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:24.112335 9488 tablet_bootstrap.cc:492] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: No bootstrap required, opened a new log
I20260928 14:58:24.112389 9488 ts_tablet_manager.cc:1397] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:24.112519 9488 raft_consensus.cc:348] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.112596 9488 raft_consensus.cc:374] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:24.112622 9488 raft_consensus.cc:729] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e90aacaa54746971e2d3d0706bb90, State: Initialized, Role: FOLLOWER
I20260928 14:58:24.112659 9488 consensus_queue.cc:260] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.112818 9488 ts_tablet_manager.cc:1428] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:24.112836 9123 heartbeater.cc:503] Master 127.1.252.252:38305 was elected leader, sending a full tablet report...
I20260928 14:58:24.113004 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.112355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:35189)
0928 14:58:24.112932 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.112969 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":81,"threads_started":1}
I20260928 14:58:24.113137 9493 tablet_bootstrap.cc:492] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Bootstrap starting.
I20260928 14:58:24.113283 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.113077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:55511)
0928 14:58:24.113203 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.113237 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":91,"threads_started":1}
I20260928 14:58:24.113296 9396 heartbeater.cc:503] Master 127.1.252.252:38305 was elected leader, sending a full tablet report...
I20260928 14:58:24.113580 9493 tablet_bootstrap.cc:654] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:24.113626 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.113476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:46507)
0928 14:58:24.113577 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.113601 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260928 14:58:24.114223 9493 tablet_bootstrap.cc:492] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: No bootstrap required, opened a new log
I20260928 14:58:24.114274 9493 ts_tablet_manager.cc:1397] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:24.114215 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.113431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:52195)
0928 14:58:24.114161 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.114191 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":714}
I20260928 14:58:24.114408 9493 raft_consensus.cc:348] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.114475 9493 raft_consensus.cc:374] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:24.114494 9493 raft_consensus.cc:729] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2d090deaf943aca44e10daf0327fa5, State: Initialized, Role: FOLLOWER
I20260928 14:58:24.114570 9493 consensus_queue.cc:260] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.114732 9493 ts_tablet_manager.cc:1428] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:24.115139 9260 heartbeater.cc:503] Master 127.1.252.252:38305 was elected leader, sending a full tablet report...
I20260928 14:58:24.115144 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.114893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:59817)
0928 14:58:24.115016 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.115106 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":81,"threads_started":1}
I20260928 14:58:24.115595 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.115500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:51995)
0928 14:58:24.115543 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.115566 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 14:58:24.148175 9494 raft_consensus.cc:482] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:24.148262 9494 raft_consensus.cc:504] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.148631 9494 leader_election.cc:288] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941), 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:24.151714 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.148863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:54715 (local address 127.1.252.194:37937)
0928 14:58:24.148915 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.148917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.148927 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:24.149004 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.149005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.149014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.149057 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.149227 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.149706 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.150971 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.151358 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.151442 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.151453 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.151607 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.151641 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:58:24.151732 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.148813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:44409 (local address 127.1.252.193:40941)
0928 14:58:24.148854 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.148857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.148910 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:58:24.149054 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.149059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.149068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.149134 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:24.149291 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.149820 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.151008 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.151362 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.151447 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.151459 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.151616 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.151654 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:58:24.151741 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.148791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37937 (local address 127.1.252.195:54715)
0928 14:58:24.148845 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.148857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.148921 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.149068 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.149069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.149070 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.149150 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.149152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.149789 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.149790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.150881 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.150884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.151441 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.151442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.151515 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.151519 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.151588 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.151628 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:58:24.151747 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.148747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40941 (local address 127.1.252.195:44409)
0928 14:58:24.148848 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.148860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:24.148938 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:24.149138 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:24.149139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:24.149141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:24.149202 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.149207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.149916 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.149917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.150917 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:24.150920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.151472 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.151473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:24.151539 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.151542 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:24.151610 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:58:24.151650 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 14:58:24.152020 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4b5b8f678405a9425874c64ba05e6" candidate_uuid: "b45e90aacaa54746971e2d3d0706bb90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2d090deaf943aca44e10daf0327fa5" is_pre_election: true
I20260928 14:58:24.152001 9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4b5b8f678405a9425874c64ba05e6" candidate_uuid: "b45e90aacaa54746971e2d3d0706bb90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f977979ca01c41698a9499eecdedc97d" is_pre_election: true
I20260928 14:58:24.152107 9214 raft_consensus.cc:2454] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45e90aacaa54746971e2d3d0706bb90 in term 0.
I20260928 14:58:24.152122 9077 raft_consensus.cc:2454] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45e90aacaa54746971e2d3d0706bb90 in term 0.
I20260928 14:58:24.152258 9283 leader_election.cc:302] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2d090deaf943aca44e10daf0327fa5, b45e90aacaa54746971e2d3d0706bb90; no voters:
I20260928 14:58:24.152381 9494 raft_consensus.cc:2781] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:24.152423 9494 raft_consensus.cc:482] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:24.152442 9494 raft_consensus.cc:3037] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:24.153090 9494 raft_consensus.cc:504] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.153235 9494 leader_election.cc:288] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [CANDIDATE]: Term 1 election: Requested vote from peers f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941), 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937)
I20260928 14:58:24.153394 9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4b5b8f678405a9425874c64ba05e6" candidate_uuid: "b45e90aacaa54746971e2d3d0706bb90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f977979ca01c41698a9499eecdedc97d"
I20260928 14:58:24.153394 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4b5b8f678405a9425874c64ba05e6" candidate_uuid: "b45e90aacaa54746971e2d3d0706bb90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2d090deaf943aca44e10daf0327fa5"
I20260928 14:58:24.153467 9077 raft_consensus.cc:3037] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:24.153466 9214 raft_consensus.cc:3037] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:24.154175 9214 raft_consensus.cc:2454] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45e90aacaa54746971e2d3d0706bb90 in term 1.
I20260928 14:58:24.154280 9077 raft_consensus.cc:2454] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45e90aacaa54746971e2d3d0706bb90 in term 1.
I20260928 14:58:24.154335 9283 leader_election.cc:302] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2d090deaf943aca44e10daf0327fa5, b45e90aacaa54746971e2d3d0706bb90; no voters:
I20260928 14:58:24.154428 9494 raft_consensus.cc:2781] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:24.154570 9494 raft_consensus.cc:686] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 1 LEADER]: Becoming Leader. State: Replica: b45e90aacaa54746971e2d3d0706bb90, State: Running, Role: LEADER
I20260928 14:58:24.154623 9494 consensus_queue.cc:237] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } }
I20260928 14:58:24.154919 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.154785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:34987)
0928 14:58:24.154834 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.154862 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:58:24.155377 8944 catalog_manager.cc:5495] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 reported cstate change: term changed from 0 to 1, leader changed from <none> to b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195). New cstate: current_term: 1 leader_uuid: "b45e90aacaa54746971e2d3d0706bb90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45e90aacaa54746971e2d3d0706bb90" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:24.166714 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.164322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.1.252.195:35627)
0928 14:58:24.164374 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:24.164375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:24.164406 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:58:24.164440 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:24.164441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:24.164455 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:24.164507 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:24.164676 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.165112 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.166011 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:24.166185 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:24.166239 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:24.166462 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:24.166550 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:24.166605 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:24.166652 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 14:58:24.166669 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:58:24.167693 9214 raft_consensus.cc:1261] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Refusing update from remote peer b45e90aacaa54746971e2d3d0706bb90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:24.167791 9077 raft_consensus.cc:1261] T c3d4b5b8f678405a9425874c64ba05e6 P f977979ca01c41698a9499eecdedc97d [term 1 FOLLOWER]: Refusing update from remote peer b45e90aacaa54746971e2d3d0706bb90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:24.167981 9500 consensus_queue.cc:1035] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2d090deaf943aca44e10daf0327fa5" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:24.168087 9494 consensus_queue.cc:1035] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f977979ca01c41698a9499eecdedc97d" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:24.169076 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.168973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:50095)
0928 14:58:24.169018 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.169047 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 14:58:24.169322 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.169208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:36623)
0928 14:58:24.169257 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.169286 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 14:58:24.170689 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.170583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.194:57021)
0928 14:58:24.170618 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.170644 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 14:58:24.170838 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.170746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.193:38857)
0928 14:58:24.170784 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.170809 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 14:58:24.170914 9506 mvcc.cc:205] Tried to move back new op lower bound from 7334328337068879872 to 7334328337017843712. Current Snapshot: MvccSnapshot[applied={T|T < 7334328337068879872}]
I20260928 14:58:24.171344 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.170652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:44901 (local address 127.1.252.195:36027)
0928 14:58:24.171285 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.171318 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:44901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618}
I20260928 14:58:24.172900 8943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:24.172995 8943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:24.174422 8943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2c7677e771a4fc0b39830a4304e5bea: Sending DeleteTablet for 3 replicas of tablet c3d4b5b8f678405a9425874c64ba05e6
I20260928 14:58:24.174715 9194 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4b5b8f678405a9425874c64ba05e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:24 UTC) from {username='oryx', principal='oryx/127.1.252.252@KRBTEST.COM'} at 127.0.0.1:59798
I20260928 14:58:24.174736 9330 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4b5b8f678405a9425874c64ba05e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:24 UTC) from {username='oryx', principal='oryx/127.1.252.252@KRBTEST.COM'} at 127.0.0.1:37796
I20260928 14:58:24.174824 9057 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4b5b8f678405a9425874c64ba05e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:24 UTC) from {username='oryx', principal='oryx/127.1.252.252@KRBTEST.COM'} at 127.0.0.1:44344
I20260928 14:58:24.174916 9514 tablet_replica.cc:330] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: stopping tablet replica
I20260928 14:58:24.174971 9514 raft_consensus.cc:2229] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:24.175021 9514 raft_consensus.cc:2258] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:24.175127 9515 tablet_replica.cc:330] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: stopping tablet replica
I20260928 14:58:24.175186 9515 raft_consensus.cc:2229] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:24.175244 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8906
I20260928 14:58:24.175269 9515 raft_consensus.cc:2258] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:24.176407 9515 ts_tablet_manager.cc:1905] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:24.178160 9515 ts_tablet_manager.cc:1918] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:24.178210 9515 log.cc:1196] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/c3d4b5b8f678405a9425874c64ba05e6
I20260928 14:58:24.178434 9515 ts_tablet_manager.cc:1939] T c3d4b5b8f678405a9425874c64ba05e6 P b45e90aacaa54746971e2d3d0706bb90: Deleting consensus metadata
I20260928 14:58:24.178440 9514 ts_tablet_manager.cc:1905] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:24.178783 8931 catalog_manager.cc:4841] TS b45e90aacaa54746971e2d3d0706bb90 (127.1.252.195:35627): tablet c3d4b5b8f678405a9425874c64ba05e6 (table test-table [id=cdd30670b41443f99b01f4ede5ecf06c]) successfully deleted
I20260928 14:58:24.179921 9514 ts_tablet_manager.cc:1918] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:24.179982 9514 log.cc:1196] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/c3d4b5b8f678405a9425874c64ba05e6
I20260928 14:58:24.180217 9514 ts_tablet_manager.cc:1939] T c3d4b5b8f678405a9425874c64ba05e6 P 5a2d090deaf943aca44e10daf0327fa5: Deleting consensus metadata
I20260928 14:58:24.180503 8928 catalog_manager.cc:4841] TS 5a2d090deaf943aca44e10daf0327fa5 (127.1.252.194:37937): tablet c3d4b5b8f678405a9425874c64ba05e6 (table test-table [id=cdd30670b41443f99b01f4ede5ecf06c]) successfully deleted
I20260928 14:58:24.191562 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:24.191463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40941 (local address 127.0.0.1:44356)
0928 14:58:24.191506 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:24.191529 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:40941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 14:58:24.191604 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 9131
W20260928 14:58:24.191707 8929 catalog_manager.cc:4584] TS f977979ca01c41698a9499eecdedc97d (127.1.252.193:40941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3d4b5b8f678405a9425874c64ba05e6: Network error: Client connection negotiation failed: client connection to 127.1.252.193:40941: connect: Connection refused (error 111)
I20260928 14:58:24.196965 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 9267
I20260928 14:58:24.202021 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8624
I20260928 14:58:24.206980 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 8812
2026-09-28T14:58:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4746 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:24 dist-test-slave-sf70 krb5kdc[9519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:24 dist-test-slave-sf70 krb5kdc[9519](info): set up 2 sockets
Sep 28 14:58:24 dist-test-slave-sf70 krb5kdc[9519](info): commencing operation
krb5kdc: starting...
W20260928 14:58:25.254002 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab.
Entry for principal oryx/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab.
I20260928 14:58:25.371438 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45925
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab
--principal=oryx/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:25.452241 9532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:25.452427 9532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:25.452450 9532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:25.453496 9532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:25.453529 9532 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:58:25.453542 9532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:25.453555 9532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:25.453567 9532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:25.453579 9532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:25.454993 9532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:45925
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.254.keytab
--principal=oryx/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:25.455160 9532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:25.455319 9532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:25.460233 9537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:25.460196 9540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:25.460220 9538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:25.460505 9532 server_base.cc:1034] running on GCE node
I20260928 14:58:25.460724 9532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:25.460960 9532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:25.462101 9532 hybrid_clock.cc:648] HybridClock initialized: now 1790607505462088 us; error 28 us; skew 500 ppm
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:25.464754 9532 init.cc:377] Logged in from keytab as oryx/127.1.252.254@KRBTEST.COM (short username oryx)
I20260928 14:58:25.465188 9532 webserver.cc:460] Webserver started at http://127.1.252.254:46205/ using document root <none> and password file <none>
I20260928 14:58:25.465390 9532 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:25.465431 9532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:25.465516 9532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:25.466554 9532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "967e29b9571a4c3287f028aa7a0905e6"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "897424c24f5c3dd7ba8f605bec8c4942"
server_key_iv: "41920769cc4559e5b67678baf2e7f815"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.466864 9532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "967e29b9571a4c3287f028aa7a0905e6"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "897424c24f5c3dd7ba8f605bec8c4942"
server_key_iv: "41920769cc4559e5b67678baf2e7f815"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.468161 9532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:58:25.468875 9547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:25.469043 9532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:25.469146 9532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "967e29b9571a4c3287f028aa7a0905e6"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "897424c24f5c3dd7ba8f605bec8c4942"
server_key_iv: "41920769cc4559e5b67678baf2e7f815"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.469218 9532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:25.481643 9532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:25.483999 9532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:25.484153 9532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:25.489147 9532 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:45925
I20260928 14:58:25.489194 9609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:45925 every 8 connection(s)
I20260928 14:58:25.489548 9532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:25.490238 9610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:25.492540 9610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Bootstrap starting.
I20260928 14:58:25.493167 9610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:25.493474 9610 log.cc:824] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:25.494158 9610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: No bootstrap required, opened a new log
I20260928 14:58:25.496102 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 9532
I20260928 14:58:25.496206 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:25.496302 9610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967e29b9571a4c3287f028aa7a0905e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45925 } }
I20260928 14:58:25.496428 2035 external_mini_cluster.cc:1428] Setting key a35e0ee8657617fd90a54a71c6a66368
I20260928 14:58:25.496448 9610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:25.496477 9610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967e29b9571a4c3287f028aa7a0905e6, State: Initialized, Role: FOLLOWER
I20260928 14:58:25.496631 9610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967e29b9571a4c3287f028aa7a0905e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45925 } }
I20260928 14:58:25.496733 9610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:25.496793 9610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:25.496845 9610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:25.497539 9610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967e29b9571a4c3287f028aa7a0905e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45925 } }
I20260928 14:58:25.497645 9610 leader_election.cc:302] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 967e29b9571a4c3287f028aa7a0905e6; no voters:
I20260928 14:58:25.497798 9610 leader_election.cc:288] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:25.497854 9615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:25.497987 9615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [term 1 LEADER]: Becoming Leader. State: Replica: 967e29b9571a4c3287f028aa7a0905e6, State: Running, Role: LEADER
I20260928 14:58:25.498055 9610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:25.498121 9615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967e29b9571a4c3287f028aa7a0905e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45925 } }
I20260928 14:58:25.498464 9615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967e29b9571a4c3287f028aa7a0905e6. Latest consensus state: current_term: 1 leader_uuid: "967e29b9571a4c3287f028aa7a0905e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967e29b9571a4c3287f028aa7a0905e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45925 } } }
I20260928 14:58:25.498476 9616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "967e29b9571a4c3287f028aa7a0905e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967e29b9571a4c3287f028aa7a0905e6" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45925 } } }
I20260928 14:58:25.498660 9616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:25.498827 9615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:25.498944 9624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:25.499447 9624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:25.501909 9624 catalog_manager.cc:1347] Generated new cluster ID: 82638c3b75024e77915b2727be107b94
I20260928 14:58:25.501964 9624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:25.507752 9624 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:25.508107 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.496923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.1.252.254:45925)
0928 14:58:25.497105 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:25.497115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:25.497149 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:58:25.497259 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:25.497266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:25.497323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:25.497475 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:25.498254 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.499751 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.500825 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.501253 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.501338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.504158 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:25.504162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:25.504163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:25.504175 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:25.506189 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.506577 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.506580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.506580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.506728 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.506949 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.506950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.506950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.507288 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:25.507449 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:25.507672 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:58:25.507813 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260928 14:58:25.508386 9624 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:25.512563 9624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Generated new TSK 0
I20260928 14:58:25.512737 9624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab.
Entry for principal oryx/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab.
I20260928 14:58:25.542951 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45925
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:25.647127 9636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:25.647351 9636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:25.647389 9636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:25.648592 9636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:25.648658 9636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:25.648689 9636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:58:25.648729 9636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:25.650203 9636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:25.650466 9636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:25.650681 9636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:25.656718 9642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:25.656836 9641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:25.656960 9636 server_base.cc:1034] running on GCE node
W20260928 14:58:25.656730 9644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:25.657191 9636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:25.657461 9636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:25.658638 9636 hybrid_clock.cc:648] HybridClock initialized: now 1790607505658619 us; error 32 us; skew 500 ppm
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:25.661266 9636 init.cc:377] Logged in from keytab as oryx/127.1.252.193@KRBTEST.COM (short username oryx)
I20260928 14:58:25.661770 9636 webserver.cc:460] Webserver started at http://127.1.252.193:36001/ using document root <none> and password file <none>
I20260928 14:58:25.661994 9636 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:25.662037 9636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:25.662175 9636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:25.663199 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "d228e9c255824229b62ff13a94e83f21"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "a15f3cb2766986704789f5f0220f46ee"
server_key_iv: "ed86c6391e519195dd17e143ef97145a"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.663552 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "d228e9c255824229b62ff13a94e83f21"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "a15f3cb2766986704789f5f0220f46ee"
server_key_iv: "ed86c6391e519195dd17e143ef97145a"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.664958 9636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:25.665720 9651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:25.665978 9636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:25.666056 9636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "d228e9c255824229b62ff13a94e83f21"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "a15f3cb2766986704789f5f0220f46ee"
server_key_iv: "ed86c6391e519195dd17e143ef97145a"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.666148 9636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:25.701745 9636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:25.709656 9636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:25.709812 9636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:25.710368 9636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:25.710409 9636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:25.710438 9636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:25.710486 9636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:25.717831 9636 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:43143
I20260928 14:58:25.717962 9766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:43143 every 8 connection(s)
I20260928 14:58:25.718230 9636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:25.718700 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 9636
I20260928 14:58:25.718813 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:25.719034 2035 external_mini_cluster.cc:1428] Setting key 8b7516985c43ac5a6da3dfda08256cc4
I20260928 14:58:25.720650 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.710907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.1.252.193:48531)
0928 14:58:25.711067 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:25.711102 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:25.711622 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:25.711913 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:25.711918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:25.712228 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:25.712424 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.712429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.713177 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.713178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.714668 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.714671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.715204 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.715205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.715352 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.715853 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:25.715873 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:25.717279 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:25.719245 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.719251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.719253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.719545 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.719770 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.719770 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.719771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.719923 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.720193 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:25.720196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:25.720304 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 14:58:25.720387 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:58:25.720502 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 14:58:25.720676 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.710927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:48531 (local address 127.1.252.254:45925)
0928 14:58:25.710984 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:25.710986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:25.711191 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:58:25.711728 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:25.711729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:25.711745 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:25.711807 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:25.712537 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.713072 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.714773 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.715107 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.715185 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.717382 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:25.717384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:25.717384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:25.717394 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:25.719162 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.719636 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.719638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.719638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.719700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.719981 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.719982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.719983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.720128 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:25.720195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:25.720444 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 14:58:25.720537 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:58:25.721454 9767 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45925
I20260928 14:58:25.721585 9767 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:25.721825 9767 heartbeater.cc:511] Master 127.1.252.254:45925 requested a full tablet report, sending...
I20260928 14:58:25.722417 9563 ts_manager.cc:194] Registered new tserver with Master: d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143)
I20260928 14:58:25.723425 9563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:48531
WARNING: no policy specified for oryx/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 14:58:25.728292 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.722285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:39313 (local address 127.1.252.254:45925)
0928 14:58:25.722319 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:25.722321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:25.722447 (+ 126us) server_negotiation.cc:407] Connection header received
0928 14:58:25.722567 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:25.722568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:25.722579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:25.722608 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:25.722969 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.723542 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.724534 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.724740 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.724823 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.725694 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:25.725696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:25.725696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:25.725702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:25.727115 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.727459 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.727461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.727461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.727512 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.727729 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.727730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.727730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.727853 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:25.727975 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:25.728132 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:58:25.728207 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:25.728298 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.722247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.1.252.193:39313)
0928 14:58:25.722362 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:25.722371 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:25.722461 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:25.722671 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:25.722671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:25.722823 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:25.722881 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.722882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.723648 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.723649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.724447 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.724449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.724824 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.724825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.724888 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.725314 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:25.725319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:25.725590 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:25.727199 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.727200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.727200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.727374 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.727594 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.727595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.727595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.727657 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.727970 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:25.727970 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:25.728015 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:58:25.728108 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:58:25.728206 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab.
Entry for principal oryx/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab.
I20260928 14:58:25.745079 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45925
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:25.825098 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:25.825325 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:25.825363 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:25.826563 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:25.826625 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:25.826659 9776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:58:25.826716 9776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:25.828260 9776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:25.828523 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:25.828750 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:25.833855 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:25.833949 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:25.833935 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:25.834062 9776 server_base.cc:1034] running on GCE node
I20260928 14:58:25.834408 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:25.834658 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:25.835811 9776 hybrid_clock.cc:648] HybridClock initialized: now 1790607505835799 us; error 28 us; skew 500 ppm
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:25.838264 9776 init.cc:377] Logged in from keytab as oryx/127.1.252.194@KRBTEST.COM (short username oryx)
I20260928 14:58:25.838642 9776 webserver.cc:460] Webserver started at http://127.1.252.194:45397/ using document root <none> and password file <none>
I20260928 14:58:25.838836 9776 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:25.838929 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:25.839083 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:25.840181 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "97d16299dd36478e89644f3b7d82ea1a"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "59f5e9de39c060946be02ec4ddd0a6a8"
server_key_iv: "54c366e44b1058275653b62e31185108"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.840549 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "97d16299dd36478e89644f3b7d82ea1a"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "59f5e9de39c060946be02ec4ddd0a6a8"
server_key_iv: "54c366e44b1058275653b62e31185108"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.841918 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:25.842746 9791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:25.842937 9776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:25.843030 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "97d16299dd36478e89644f3b7d82ea1a"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "59f5e9de39c060946be02ec4ddd0a6a8"
server_key_iv: "54c366e44b1058275653b62e31185108"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.843114 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:25.855484 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:25.857483 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:25.857643 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:25.858122 9776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:25.858187 9776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:25.858230 9776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:25.858256 9776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:25.865278 9776 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:37735
I20260928 14:58:25.865343 9906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:37735 every 8 connection(s)
I20260928 14:58:25.865707 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:25.868338 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.858625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50443 (local address 127.1.252.254:45925)
0928 14:58:25.858699 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:25.858702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:25.858856 (+ 154us) server_negotiation.cc:407] Connection header received
0928 14:58:25.859351 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:25.859353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:25.859364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:25.859447 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:25.860303 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.861137 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.862855 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.863205 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.863286 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.865746 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:25.865747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:25.865747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:25.865771 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:25.867225 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.867586 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.867586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.867587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.867637 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.867805 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.867806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.867806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.867963 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:25.868015 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:25.868219 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 14:58:25.868277 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:58:25.868498 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.858587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.1.252.194:50443)
0928 14:58:25.858725 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:25.858765 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:25.859259 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:25.859516 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:25.859519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:25.859880 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:25.860190 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.860197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.861233 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.861234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.862734 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.862737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.863315 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.863316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.863439 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.863969 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:25.863978 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:25.865634 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:25.867295 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.867299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.867301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.867519 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.867678 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.867679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.867679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.867758 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.868010 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:25.868013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:25.868117 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 14:58:25.868204 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:58:25.868316 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260928 14:58:25.869045 9907 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45925
I20260928 14:58:25.869153 9907 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:25.869346 9907 heartbeater.cc:511] Master 127.1.252.254:45925 requested a full tablet report, sending...
I20260928 14:58:25.869431 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 9776
I20260928 14:58:25.869547 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:25.869727 9563 ts_manager.cc:194] Registered new tserver with Master: 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735)
I20260928 14:58:25.869776 2035 external_mini_cluster.cc:1428] Setting key 73dfc3f413ea4abe41ca04eef7fa8c82
I20260928 14:58:25.870214 9563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:50443
I20260928 14:58:25.876075 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.870127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50023 (local address 127.1.252.254:45925)
0928 14:58:25.870161 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:25.870163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:25.870501 (+ 338us) server_negotiation.cc:407] Connection header received
0928 14:58:25.870560 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:25.870560 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:25.870571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:25.870628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:25.871017 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.871452 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.872389 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.872609 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.872682 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.873680 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:25.873682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:25.873682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:25.873689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:25.874927 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.875261 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.875262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.875262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.875319 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:25.875518 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:25.875519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:25.875519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:25.875661 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:25.875734 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:25.875910 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:58:25.875990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:25.876078 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:25.870190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.1.252.194:50023)
0928 14:58:25.870439 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:25.870454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:25.870512 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:25.870632 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:25.870633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:25.870858 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:25.870927 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.870929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.871550 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.871551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.872278 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:25.872280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:25.872682 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:25.872683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:25.872749 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:25.873203 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:25.873213 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:25.873603 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:25.875008 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.875009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.875009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.875180 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.875385 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:25.875386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:25.875386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:25.875444 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:25.875738 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:25.875739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:25.875784 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:58:25.875869 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:58:25.875987 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab.
Entry for principal oryx/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab.
I20260928 14:58:25.895237 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45925
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:25.978857 9916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:25.979075 9916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:25.979099 9916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:25.980321 9916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:25.980362 9916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:25.980388 9916 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:58:25.980429 9916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:25.981920 9916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.252.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:25.982154 9916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:25.982414 9916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:25.987357 9924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:25.987533 9916 server_base.cc:1034] running on GCE node
W20260928 14:58:25.987637 9921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:25.987363 9922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:25.988900 9916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:25.989166 9916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:25.990392 9916 hybrid_clock.cc:648] HybridClock initialized: now 1790607505990372 us; error 31 us; skew 500 ppm
Sep 28 14:58:25 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:25.993075 9916 init.cc:377] Logged in from keytab as oryx/127.1.252.195@KRBTEST.COM (short username oryx)
I20260928 14:58:25.993458 9916 webserver.cc:460] Webserver started at http://127.1.252.195:40483/ using document root <none> and password file <none>
I20260928 14:58:25.993646 9916 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:25.993686 9916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:25.993781 9916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:25.994798 9916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "b1f222e0c392457ba13d4371e0d41568"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "e71db2cf2857a640057323e51750d8da"
server_key_iv: "2e64cd50998a4e30c20b58d91203a9ad"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.995091 9916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "b1f222e0c392457ba13d4371e0d41568"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "e71db2cf2857a640057323e51750d8da"
server_key_iv: "2e64cd50998a4e30c20b58d91203a9ad"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.996376 9916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:25.997020 9931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:25.997200 9916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:25.997268 9916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "b1f222e0c392457ba13d4371e0d41568"
format_stamp: "Formatted at 2026-09-28 14:58:25 on dist-test-slave-sf70"
server_key: "e71db2cf2857a640057323e51750d8da"
server_key_iv: "2e64cd50998a4e30c20b58d91203a9ad"
server_key_version: "encryptionkey@0"
I20260928 14:58:25.997368 9916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:26.004834 9916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:26.006459 9916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:26.006609 9916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:26.007055 9916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:26.007088 9916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.007144 9916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:26.007180 9916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:26.014233 9916 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35911
I20260928 14:58:26.014302 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35911 every 8 connection(s)
I20260928 14:58:26.014580 9916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:26.017448 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.007547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:60073 (local address 127.1.252.254:45925)
0928 14:58:26.007618 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.007621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.007770 (+ 149us) server_negotiation.cc:407] Connection header received
0928 14:58:26.008414 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.008415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.008428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.008502 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.009272 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.010102 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.011647 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.012017 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.012100 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.014690 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:26.014691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:26.014692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:26.014700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:26.016110 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.016502 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.016504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.016504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.016572 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.016823 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.016824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.016825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.016943 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:26.017013 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.017277 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.017358 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:58:26.017625 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.007513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.1.252.195:60073)
0928 14:58:26.007649 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.007686 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.008318 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.008584 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.008592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.008898 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:26.009168 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.009173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.010233 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.010235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.011558 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.011560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.012119 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.012120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.012262 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.012893 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:26.012901 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:26.014593 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:26.016187 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:26.016193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:26.016196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:26.016412 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:26.016639 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:26.016640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:26.016640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:26.016749 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:26.017015 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:26.017022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:26.017151 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.017254 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.017409 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260928 14:58:26.018090 10047 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45925
I20260928 14:58:26.018196 10047 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:26.018366 10047 heartbeater.cc:511] Master 127.1.252.254:45925 requested a full tablet report, sending...
I20260928 14:58:26.018745 9563 ts_manager.cc:194] Registered new tserver with Master: b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
I20260928 14:58:26.019183 9563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:60073
I20260928 14:58:26.019959 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 9916
I20260928 14:58:26.020083 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:26.020257 2035 external_mini_cluster.cc:1428] Setting key cd3798e5027d8c6a2f5909cf3d7af2f0
I20260928 14:58:26.021047 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:58:26.025157 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.019137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:50903 (local address 127.1.252.254:45925)
0928 14:58:26.019178 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.019180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.019241 (+ 61us) server_negotiation.cc:407] Connection header received
0928 14:58:26.019321 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.019321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.019330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.019370 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.019831 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.020508 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.021316 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.021515 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.021586 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.022565 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:26.022566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:26.022566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:26.022573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:26.024032 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.024379 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.024380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.024381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.024433 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.024665 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.024666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.024666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.024772 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:26.024842 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.024999 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.025070 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:26.025170 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.019055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.1.252.195:50903)
0928 14:58:26.019182 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.019193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.019255 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.019440 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.019441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.019633 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:26.019690 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.019691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.020581 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.020582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.021223 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.021225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.021595 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.021596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.021660 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.022161 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:26.022166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:26.022458 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:26.024122 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:26.024123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:26.024123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:26.024294 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:26.024502 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:26.024502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:26.024503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:26.024577 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:26.024836 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:26.024837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:26.024887 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.024974 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.025073 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:58:26.041740 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:26.036584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.1.252.254:45925)
0928 14:58:26.036629 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.036630 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.036640 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:26.036698 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.036699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.036709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.036780 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.037351 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.037818 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.038566 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.038753 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.038853 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.041685 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41136: BlockingRecv error: recv got EOF from 127.0.0.1:41136 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:26.049188 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.042588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.1.252.254:45925)
0928 14:58:26.042637 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.042638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.042745 (+ 107us) server_negotiation.cc:407] Connection header received
0928 14:58:26.042789 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.042790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.042798 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.042843 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.043242 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.043679 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.044456 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.044634 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.044692 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.046676 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:26.046677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:26.046678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:26.046684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:26.048147 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.048445 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.048446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.048446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.048511 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.048785 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.048785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.048786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.048887 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:26.048962 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.049051 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.049104 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:58:26.050729 9563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:26.052443 9563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:26.063189 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.058111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43143 (local address 127.0.0.1:35210)
0928 14:58:26.058271 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.058314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.058380 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.058913 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.058916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.058919 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.058997 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.058999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.060457 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.060458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.061973 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.061975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.062938 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.062997 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.063004 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.063056 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.063086 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":46,"thread_start_us":45,"threads_started":1}
I20260928 14:58:26.063199 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.058351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.1.252.193:43143)
0928 14:58:26.058734 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.058738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.058749 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:58:26.058789 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.058791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.058807 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.059066 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.059119 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.060357 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062071 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.062840 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062934 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.063011 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.063067 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.063105 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":79,"threads_started":1}
I20260928 14:58:26.063433 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.058119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35911 (local address 127.0.0.1:59342)
0928 14:58:26.058682 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.058691 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.058750 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.059014 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.059015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.059017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.059208 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.059210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.060564 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.060565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.061956 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.061959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.062803 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.062896 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.062909 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.062977 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.063036 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":59,"thread_start_us":102,"threads_started":1}
I20260928 14:58:26.063666 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.058387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.1.252.195:35911)
0928 14:58:26.058580 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.058585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.058809 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:58:26.058855 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.058857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.058872 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.058944 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.059268 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.060475 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062042 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.062703 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062800 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.062878 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.063519 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.063556 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260928 14:58:26.064051 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.058568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37735 (local address 127.0.0.1:51482)
0928 14:58:26.059970 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.059980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.060038 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.060284 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.060285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.060287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.060322 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.060323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.061657 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.061658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.062732 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.062735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.063768 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.063769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.063834 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.063838 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.063929 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.063971 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":74,"threads_started":1}
I20260928 14:58:26.064095 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.058685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.1.252.194:37735)
0928 14:58:26.059128 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.059133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.060101 (+ 968us) server_negotiation.cc:407] Connection header received
0928 14:58:26.060143 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.060145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.060161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.060224 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.060399 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.061571 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.062837 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.063589 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.063652 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.063732 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.063949 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.063994 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":124,"threads_started":1}
I20260928 14:58:26.064013 9684 tablet_service.cc:1467] Processing CreateTablet for tablet 221a630ff98341ffb45200dbbd048079 (DEFAULT_TABLE table=test-table [id=2988f58d6bbf42f7bd11249db8cd7749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.064347 9975 tablet_service.cc:1467] Processing CreateTablet for tablet 221a630ff98341ffb45200dbbd048079 (DEFAULT_TABLE table=test-table [id=2988f58d6bbf42f7bd11249db8cd7749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.064704 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221a630ff98341ffb45200dbbd048079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.064815 9841 tablet_service.cc:1467] Processing CreateTablet for tablet 221a630ff98341ffb45200dbbd048079 (DEFAULT_TABLE table=test-table [id=2988f58d6bbf42f7bd11249db8cd7749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.068162 10075 tablet_bootstrap.cc:492] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Bootstrap starting.
I20260928 14:58:26.068825 10075 tablet_bootstrap.cc:654] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.069170 10075 log.cc:824] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:26.070019 10075 tablet_bootstrap.cc:492] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: No bootstrap required, opened a new log
I20260928 14:58:26.070071 10075 ts_tablet_manager.cc:1397] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:26.071421 10075 raft_consensus.cc:348] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.071548 10075 raft_consensus.cc:374] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.071569 10075 raft_consensus.cc:729] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.071683 10075 consensus_queue.cc:260] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.071969 10075 ts_tablet_manager.cc:1428] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:26.071995 10047 heartbeater.cc:503] Master 127.1.252.254:45925 was elected leader, sending a full tablet report...
I20260928 14:58:26.077127 9841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221a630ff98341ffb45200dbbd048079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.077127 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221a630ff98341ffb45200dbbd048079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.081211 10079 tablet_bootstrap.cc:492] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Bootstrap starting.
I20260928 14:58:26.081300 10078 tablet_bootstrap.cc:492] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Bootstrap starting.
I20260928 14:58:26.081780 10079 tablet_bootstrap.cc:654] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.081837 10078 tablet_bootstrap.cc:654] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.082103 10078 log.cc:824] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:26.082110 10079 log.cc:824] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:26.082829 10078 tablet_bootstrap.cc:492] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: No bootstrap required, opened a new log
I20260928 14:58:26.082829 10079 tablet_bootstrap.cc:492] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: No bootstrap required, opened a new log
I20260928 14:58:26.082912 10078 ts_tablet_manager.cc:1397] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:26.082928 10079 ts_tablet_manager.cc:1397] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:26.084326 10079 raft_consensus.cc:348] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.084326 10078 raft_consensus.cc:348] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.084460 10079 raft_consensus.cc:374] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.084481 10079 raft_consensus.cc:729] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d228e9c255824229b62ff13a94e83f21, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.084483 10078 raft_consensus.cc:374] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.084537 10078 raft_consensus.cc:729] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.084625 10079 consensus_queue.cc:260] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.084659 10078 consensus_queue.cc:260] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.084928 10079 ts_tablet_manager.cc:1428] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:26.084968 10078 ts_tablet_manager.cc:1428] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:26.084954 9767 heartbeater.cc:503] Master 127.1.252.254:45925 was elected leader, sending a full tablet report...
I20260928 14:58:26.085239 9907 heartbeater.cc:503] Master 127.1.252.254:45925 was elected leader, sending a full tablet report...
I20260928 14:58:26.097487 10077 raft_consensus.cc:482] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:26.097611 10077 raft_consensus.cc:504] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.098003 10077 leader_election.cc:288] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735)
I20260928 14:58:26.101263 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.098257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:45917 (local address 127.1.252.194:37735)
0928 14:58:26.098300 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.098302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.098312 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:26.098338 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.098339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.098348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.098414 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.098584 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.099391 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.100449 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.100917 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.100977 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.100984 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.101157 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.101190 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:26.101276 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.098178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37735 (local address 127.1.252.195:45917)
0928 14:58:26.098230 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.098243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.098285 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.098418 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.098419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.098421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.098510 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.098512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.099466 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.099467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.100371 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.100373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.100997 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.100998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.101086 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.101090 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.101144 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.101184 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260928 14:58:26.101421 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.098235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:35879 (local address 127.1.252.193:43143)
0928 14:58:26.098280 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.098282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.098430 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:58:26.098549 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.098551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.098562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.098618 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.098783 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.099469 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.100627 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.101051 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.101126 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.101135 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.101300 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.101344 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:58:26.101408 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.098200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43143 (local address 127.1.252.195:35879)
0928 14:58:26.098351 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.098368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.098444 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.098626 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.098627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.098628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.098713 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.098714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.099535 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.099536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.100382 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.100384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.101153 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.101154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.101223 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.101227 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.101284 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.101325 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260928 14:58:26.101651 9861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221a630ff98341ffb45200dbbd048079" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d16299dd36478e89644f3b7d82ea1a" is_pre_election: true
I20260928 14:58:26.101801 9861 raft_consensus.cc:2454] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 0.
I20260928 14:58:26.101915 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221a630ff98341ffb45200dbbd048079" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21" is_pre_election: true
I20260928 14:58:26.102022 9933 leader_election.cc:302] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, b1f222e0c392457ba13d4371e0d41568; no voters:
I20260928 14:58:26.102053 9721 raft_consensus.cc:2454] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 0.
I20260928 14:58:26.102139 10077 raft_consensus.cc:2781] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:26.102196 10077 raft_consensus.cc:482] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:26.102221 10077 raft_consensus.cc:3037] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.103082 10077 raft_consensus.cc:504] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.103238 10077 leader_election.cc:288] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 election: Requested vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735)
I20260928 14:58:26.103358 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221a630ff98341ffb45200dbbd048079" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21"
I20260928 14:58:26.103358 9861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221a630ff98341ffb45200dbbd048079" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d16299dd36478e89644f3b7d82ea1a"
I20260928 14:58:26.103433 9861 raft_consensus.cc:3037] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.103435 9721 raft_consensus.cc:3037] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.104135 9861 raft_consensus.cc:2454] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 1.
I20260928 14:58:26.104136 9721 raft_consensus.cc:2454] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 1.
I20260928 14:58:26.104305 9933 leader_election.cc:302] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, b1f222e0c392457ba13d4371e0d41568; no voters:
I20260928 14:58:26.104411 10077 raft_consensus.cc:2781] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:26.104569 10077 raft_consensus.cc:686] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 1 LEADER]: Becoming Leader. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Running, Role: LEADER
I20260928 14:58:26.104671 10077 consensus_queue.cc:237] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } }
I20260928 14:58:26.105434 9564 catalog_manager.cc:5495] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1f222e0c392457ba13d4371e0d41568 (127.1.252.195). New cstate: current_term: 1 leader_uuid: "b1f222e0c392457ba13d4371e0d41568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260928 14:58:26.116255 9908 tablet.cc:2369] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:26.119952 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.117470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.1.252.195:35911)
0928 14:58:26.117511 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.117513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.117521 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:26.117604 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.117605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.117620 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.117730 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:26.117890 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.118328 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.119156 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.119343 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.119419 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.119601 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.119691 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.119773 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.119824 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.119845 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:58:26.121606 9861 raft_consensus.cc:1261] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Refusing update from remote peer b1f222e0c392457ba13d4371e0d41568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.121620 9721 raft_consensus.cc:1261] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Refusing update from remote peer b1f222e0c392457ba13d4371e0d41568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.122025 10085 consensus_queue.cc:1035] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.122191 10077 consensus_queue.cc:1035] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.124269 10088 mvcc.cc:205] Tried to move back new op lower bound from 7334328345071775744 to 7334328345005449216. Current Snapshot: MvccSnapshot[applied={T|T < 7334328345071775744}]
I20260928 14:58:26.125103 10090 mvcc.cc:205] Tried to move back new op lower bound from 7334328345071775744 to 7334328345005449216. Current Snapshot: MvccSnapshot[applied={T|T < 7334328345071775744 or (T in {7334328345071775744})}]
I20260928 14:58:26.125133 10089 mvcc.cc:205] Tried to move back new op lower bound from 7334328345071775744 to 7334328345005449216. Current Snapshot: MvccSnapshot[applied={T|T < 7334328345071775744}]
I20260928 14:58:26.127065 9564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.127177 9564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.128029 9564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Sending DeleteTablet for 3 replicas of tablet 221a630ff98341ffb45200dbbd048079
I20260928 14:58:26.128324 9975 tablet_service.cc:1514] Processing DeleteTablet for tablet 221a630ff98341ffb45200dbbd048079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59342
I20260928 14:58:26.128366 9684 tablet_service.cc:1514] Processing DeleteTablet for tablet 221a630ff98341ffb45200dbbd048079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260928 14:58:26.128552 9841 tablet_service.cc:1514] Processing DeleteTablet for tablet 221a630ff98341ffb45200dbbd048079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:51482
I20260928 14:58:26.128847 10099 tablet_replica.cc:330] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: stopping tablet replica
I20260928 14:58:26.128865 10100 tablet_replica.cc:330] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: stopping tablet replica
I20260928 14:58:26.128964 10099 raft_consensus.cc:2229] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.128911 9564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:26.129057 9564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:26.129077 10099 raft_consensus.cc:2258] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.129246 10100 raft_consensus.cc:2229] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.129563 10100 raft_consensus.cc:2258] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.130039 10100 ts_tablet_manager.cc:1905] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.130501 9841 tablet_service.cc:1467] Processing CreateTablet for tablet dda75cb559f04f3194e3b17695ee0902 (DEFAULT_TABLE table=test-table [id=103582e2d0c4422c8f53c971e299040b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.130594 9684 tablet_service.cc:1467] Processing CreateTablet for tablet dda75cb559f04f3194e3b17695ee0902 (DEFAULT_TABLE table=test-table [id=103582e2d0c4422c8f53c971e299040b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.130620 9841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda75cb559f04f3194e3b17695ee0902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.130653 9975 tablet_service.cc:1467] Processing CreateTablet for tablet dda75cb559f04f3194e3b17695ee0902 (DEFAULT_TABLE table=test-table [id=103582e2d0c4422c8f53c971e299040b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.130704 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda75cb559f04f3194e3b17695ee0902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.130764 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda75cb559f04f3194e3b17695ee0902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.132023 10078 tablet_bootstrap.cc:492] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Bootstrap starting.
I20260928 14:58:26.132052 10079 tablet_bootstrap.cc:492] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Bootstrap starting.
I20260928 14:58:26.132458 10079 tablet_bootstrap.cc:654] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.132480 10078 tablet_bootstrap.cc:654] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.132758 10099 ts_tablet_manager.cc:1905] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.132815 10075 tablet_bootstrap.cc:492] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Bootstrap starting.
I20260928 14:58:26.133136 10079 tablet_bootstrap.cc:492] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: No bootstrap required, opened a new log
I20260928 14:58:26.133181 10079 ts_tablet_manager.cc:1397] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:26.133283 10075 tablet_bootstrap.cc:654] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.133322 10079 raft_consensus.cc:348] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.133394 10079 raft_consensus.cc:374] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.133414 10079 raft_consensus.cc:729] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d228e9c255824229b62ff13a94e83f21, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.133463 10079 consensus_queue.cc:260] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.133549 10079 ts_tablet_manager.cc:1428] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.134747 10099 ts_tablet_manager.cc:1918] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:26.134828 10099 log.cc:1196] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/221a630ff98341ffb45200dbbd048079
I20260928 14:58:26.135120 10099 ts_tablet_manager.cc:1939] T 221a630ff98341ffb45200dbbd048079 P d228e9c255824229b62ff13a94e83f21: Deleting consensus metadata
I20260928 14:58:26.135349 10100 ts_tablet_manager.cc:1918] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:26.135422 10100 log.cc:1196] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/221a630ff98341ffb45200dbbd048079
I20260928 14:58:26.135442 10078 tablet_bootstrap.cc:492] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: No bootstrap required, opened a new log
I20260928 14:58:26.135486 10078 ts_tablet_manager.cc:1397] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:58:26.135607 10075 tablet_bootstrap.cc:492] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: No bootstrap required, opened a new log
I20260928 14:58:26.135650 10075 ts_tablet_manager.cc:1397] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:58:26.135627 10078 raft_consensus.cc:348] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.135695 10078 raft_consensus.cc:374] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.135717 10078 raft_consensus.cc:729] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.135772 10078 consensus_queue.cc:260] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.135805 9548 catalog_manager.cc:4841] TS d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143): tablet 221a630ff98341ffb45200dbbd048079 (table test-table [id=2988f58d6bbf42f7bd11249db8cd7749]) successfully deleted
I20260928 14:58:26.135850 10078 ts_tablet_manager.cc:1428] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.135807 10075 raft_consensus.cc:348] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.135910 10075 raft_consensus.cc:374] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.135941 10075 raft_consensus.cc:729] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.135984 10075 consensus_queue.cc:260] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.136096 10075 ts_tablet_manager.cc:1428] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:26.136757 10100 ts_tablet_manager.cc:1939] T 221a630ff98341ffb45200dbbd048079 P 97d16299dd36478e89644f3b7d82ea1a: Deleting consensus metadata
I20260928 14:58:26.137025 10098 tablet_replica.cc:330] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: stopping tablet replica
I20260928 14:58:26.137143 10098 raft_consensus.cc:2229] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:26.137193 9549 catalog_manager.cc:4841] TS 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735): tablet 221a630ff98341ffb45200dbbd048079 (table test-table [id=2988f58d6bbf42f7bd11249db8cd7749]) successfully deleted
I20260928 14:58:26.137279 10098 raft_consensus.cc:2258] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.137693 10098 ts_tablet_manager.cc:1905] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.139035 10098 ts_tablet_manager.cc:1918] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:26.139097 10098 log.cc:1196] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/221a630ff98341ffb45200dbbd048079
I20260928 14:58:26.139287 10098 ts_tablet_manager.cc:1939] T 221a630ff98341ffb45200dbbd048079 P b1f222e0c392457ba13d4371e0d41568: Deleting consensus metadata
I20260928 14:58:26.139814 9551 catalog_manager.cc:4841] TS b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911): tablet 221a630ff98341ffb45200dbbd048079 (table test-table [id=2988f58d6bbf42f7bd11249db8cd7749]) successfully deleted
I20260928 14:58:26.215715 10083 raft_consensus.cc:482] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:26.215821 10083 raft_consensus.cc:504] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.216420 10083 leader_election.cc:288] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
W20260928 14:58:26.218967 9768 tablet.cc:2369] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:26.219877 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.216712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:37703 (local address 127.1.252.195:35911)
0928 14:58:26.216758 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.216760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.216769 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:26.216798 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.216799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.216807 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.216866 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.217086 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.217844 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.218969 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.219523 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.219596 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.219605 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.219772 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.219805 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:58:26.219878 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.216605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35911 (local address 127.1.252.194:37703)
0928 14:58:26.216673 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.216685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.216729 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.216907 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.216908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.216911 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.216991 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.216993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.217938 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.217939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.218866 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.218870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.219612 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.219613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.219688 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.219693 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.219753 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.219793 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:58:26.220209 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda75cb559f04f3194e3b17695ee0902" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f222e0c392457ba13d4371e0d41568" is_pre_election: true
I20260928 14:58:26.220350 10001 raft_consensus.cc:2454] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 0.
I20260928 14:58:26.220412 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.216740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50145 (local address 127.1.252.193:43143)
0928 14:58:26.216786 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.216788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.216993 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:58:26.217052 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.217053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.217063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.217114 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:26.217265 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.217975 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.219474 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.220022 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.220092 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.220101 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.220311 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.220353 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:58:26.220428 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.216608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43143 (local address 127.1.252.194:50145)
0928 14:58:26.216914 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.216923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.216998 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.217116 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.217117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.217118 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:26.217193 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.217195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.218069 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.218070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.219380 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.219383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.220136 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.220137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.220205 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.220210 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.220291 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.220341 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":175,"thread_start_us":75,"threads_started":1}
I20260928 14:58:26.220525 9795 leader_election.cc:302] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, b1f222e0c392457ba13d4371e0d41568; no voters:
I20260928 14:58:26.220749 10083 raft_consensus.cc:2781] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:26.220789 10083 raft_consensus.cc:482] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:26.220780 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda75cb559f04f3194e3b17695ee0902" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21" is_pre_election: true
I20260928 14:58:26.220839 10083 raft_consensus.cc:3037] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.220861 9721 raft_consensus.cc:2454] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 0.
I20260928 14:58:26.221645 10083 raft_consensus.cc:504] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.221783 10083 leader_election.cc:288] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 election: Requested vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
I20260928 14:58:26.221904 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda75cb559f04f3194e3b17695ee0902" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21"
I20260928 14:58:26.221990 9721 raft_consensus.cc:3037] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.222157 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda75cb559f04f3194e3b17695ee0902" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f222e0c392457ba13d4371e0d41568"
I20260928 14:58:26.222263 10001 raft_consensus.cc:3037] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.222817 9721 raft_consensus.cc:2454] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 1.
I20260928 14:58:26.222918 10001 raft_consensus.cc:2454] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 1.
I20260928 14:58:26.222993 9792 leader_election.cc:302] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.223062 10083 raft_consensus.cc:2781] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:26.223189 10083 raft_consensus.cc:686] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 1 LEADER]: Becoming Leader. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Running, Role: LEADER
I20260928 14:58:26.223263 10083 consensus_queue.cc:237] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.224015 9563 catalog_manager.cc:5495] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194). New cstate: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.254@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:26.258898 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.252632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45925 (local address 127.0.0.1:41154)
0928 14:58:26.252818 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.252829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.252897 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.253021 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.253022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.253320 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:26.253369 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.253371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.253987 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.253988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.254515 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.254516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.254830 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.254831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.254872 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.255342 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:26.255347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:26.256663 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:26.258011 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:26.258018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:26.258020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:26.258206 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:26.258374 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:26.258375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:26.258375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:26.258428 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:26.258622 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:26.258629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:26.258667 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.258716 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.258820 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 14:58:26.258949 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.252698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.1.252.254:45925)
0928 14:58:26.252792 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.252794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.252884 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:58:26.252943 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.252943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.252952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.253003 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.253443 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.253922 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.254588 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.254756 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.254803 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.256742 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:26.256743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:26.256744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:26.256750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:26.257944 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.258270 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.258271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.258272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.258318 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.258480 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.258481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.258481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.258572 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:26.258619 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.258750 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.258863 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":68}
I20260928 14:58:26.260830 9563 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:41154:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:58:26.261003 9563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:26.262383 9975 tablet_service.cc:1467] Processing CreateTablet for tablet f737d44241b048359cf508578307d9cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0dd9c5ef4d5542c289076cde1fc5560e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:26.262367 9841 tablet_service.cc:1467] Processing CreateTablet for tablet f737d44241b048359cf508578307d9cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0dd9c5ef4d5542c289076cde1fc5560e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:26.262382 9684 tablet_service.cc:1467] Processing CreateTablet for tablet f737d44241b048359cf508578307d9cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0dd9c5ef4d5542c289076cde1fc5560e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:58:26.262557 9841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f737d44241b048359cf508578307d9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.262557 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f737d44241b048359cf508578307d9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.262557 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f737d44241b048359cf508578307d9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.263914 10079 tablet_bootstrap.cc:492] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: Bootstrap starting.
I20260928 14:58:26.264022 10078 tablet_bootstrap.cc:492] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: Bootstrap starting.
I20260928 14:58:26.264379 10079 tablet_bootstrap.cc:654] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.264424 10078 tablet_bootstrap.cc:654] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.264761 10075 tablet_bootstrap.cc:492] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: Bootstrap starting.
I20260928 14:58:26.265090 10075 tablet_bootstrap.cc:654] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.265103 10079 tablet_bootstrap.cc:492] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: No bootstrap required, opened a new log
I20260928 14:58:26.265149 10079 ts_tablet_manager.cc:1397] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:26.265316 10079 raft_consensus.cc:348] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.265394 10079 raft_consensus.cc:374] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260928 14:58:26.265386 10048 tablet.cc:2369] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:26.265417 10079 raft_consensus.cc:729] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d228e9c255824229b62ff13a94e83f21, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.265456 10079 consensus_queue.cc:260] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.265554 10079 ts_tablet_manager.cc:1428] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:26.265601 10082 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.265688 10075 tablet_bootstrap.cc:492] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: No bootstrap required, opened a new log
I20260928 14:58:26.265722 10082 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.265730 10075 ts_tablet_manager.cc:1397] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:26.265872 10075 raft_consensus.cc:348] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.265939 10075 raft_consensus.cc:374] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.265957 10075 raft_consensus.cc:729] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.266019 10075 consensus_queue.cc:260] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.266119 10075 ts_tablet_manager.cc:1428] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.266148 10095 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.266212 10095 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.266417 10078 tablet_bootstrap.cc:492] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: No bootstrap required, opened a new log
I20260928 14:58:26.266458 10078 ts_tablet_manager.cc:1397] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:58:26.266780 10078 raft_consensus.cc:348] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.266839 10078 raft_consensus.cc:374] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.266852 10078 raft_consensus.cc:729] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.266909 10078 consensus_queue.cc:260] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.267005 10078 ts_tablet_manager.cc:1428] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.267043 10083 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.267112 10083 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.308192 10083 raft_consensus.cc:482] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:26.308283 10083 raft_consensus.cc:504] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.308456 10083 leader_election.cc:288] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
I20260928 14:58:26.308631 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f737d44241b048359cf508578307d9cf" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21" is_pre_election: true
I20260928 14:58:26.308708 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f737d44241b048359cf508578307d9cf" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f222e0c392457ba13d4371e0d41568" is_pre_election: true
I20260928 14:58:26.308772 9721 raft_consensus.cc:2454] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 0.
I20260928 14:58:26.308805 10001 raft_consensus.cc:2454] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 0.
I20260928 14:58:26.308997 9792 leader_election.cc:302] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.309110 10083 raft_consensus.cc:2781] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:26.309165 10083 raft_consensus.cc:482] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:26.309190 10083 raft_consensus.cc:3037] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.309865 10083 raft_consensus.cc:504] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.310022 10083 leader_election.cc:288] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 election: Requested vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
I20260928 14:58:26.310176 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f737d44241b048359cf508578307d9cf" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f222e0c392457ba13d4371e0d41568"
I20260928 14:58:26.310189 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f737d44241b048359cf508578307d9cf" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21"
I20260928 14:58:26.310252 9721 raft_consensus.cc:3037] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.310276 10001 raft_consensus.cc:3037] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.310865 9721 raft_consensus.cc:2454] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 1.
I20260928 14:58:26.310973 10001 raft_consensus.cc:2454] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 1.
I20260928 14:58:26.311069 9792 leader_election.cc:302] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.311174 10083 raft_consensus.cc:2781] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:26.311234 10083 raft_consensus.cc:686] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [term 1 LEADER]: Becoming Leader. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Running, Role: LEADER
I20260928 14:58:26.311277 10083 consensus_queue.cc:237] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.311626 10105 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: TxnStatusTablet state changed. Reason: New leader 97d16299dd36478e89644f3b7d82ea1a. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.311718 10105 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:26.311975 10117 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:58:26.311944 9564 catalog_manager.cc:5495] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194). New cstate: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:26.312072 10117 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:58:26.325589 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.322600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37735 (local address 127.0.0.1:51488)
0928 14:58:26.322672 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.322684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.322743 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.322920 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.322921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.322923 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:26.322973 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.322975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.323959 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.323960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.324523 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.324524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.325034 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.325036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.325091 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.325105 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.325433 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.325442 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.325491 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.325537 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:58:26.325589 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.322652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.1.252.194:37735)
0928 14:58:26.322690 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.322692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.322718 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:58:26.322817 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.322818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.322836 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.322904 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:26.323047 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.323816 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.324606 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.324926 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.324999 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.325213 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.325332 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.325414 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.325496 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.325518 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:58:26.326871 9721 raft_consensus.cc:1261] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Refusing update from remote peer 97d16299dd36478e89644f3b7d82ea1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.326968 10001 raft_consensus.cc:1261] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Refusing update from remote peer 97d16299dd36478e89644f3b7d82ea1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.327160 10083 consensus_queue.cc:1035] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.327252 10105 consensus_queue.cc:1035] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.327823 10082 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: TxnStatusTablet state changed. Reason: New leader 97d16299dd36478e89644f3b7d82ea1a. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.327934 10082 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.327873 10095 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: TxnStatusTablet state changed. Reason: New leader 97d16299dd36478e89644f3b7d82ea1a. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.327986 10095 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.328734 10105 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.328814 10105 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:26.329006 10095 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.329097 10095 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P b1f222e0c392457ba13d4371e0d41568: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.329594 10082 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.329617 10083 tablet_replica.cc:439] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } } }
I20260928 14:58:26.329684 10083 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P 97d16299dd36478e89644f3b7d82ea1a: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:58:26.329684 10082 tablet_replica.cc:442] T f737d44241b048359cf508578307d9cf P d228e9c255824229b62ff13a94e83f21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:58:26.332967 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.330366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.1.252.194:37735)
0928 14:58:26.330406 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.330407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.330419 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:58:26.330504 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.330505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.330514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.330564 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:26.330660 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.331347 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.332159 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.332460 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.332532 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.332697 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.332729 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.332780 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.332883 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.332902 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:58:26.337817 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.335411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:41955 (local address 127.1.252.194:37735)
0928 14:58:26.335452 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.335453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.335482 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:58:26.335507 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.335507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.335514 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.335550 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:26.335696 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.336177 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.337222 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.337396 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.337447 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.337627 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.337649 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.337690 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.337758 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.337770 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:58:26.337898 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.335327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37735 (local address 127.1.252.194:41955)
0928 14:58:26.335403 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.335413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.335447 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.335573 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:26.335575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:26.335576 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:58:26.335630 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.335631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.336249 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.336250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.337139 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:26.337141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.337467 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.337468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:26.337533 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.337547 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.337710 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.337712 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:26.337759 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 14:58:26.337800 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:58:26.340000 10001 raft_consensus.cc:1261] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Refusing update from remote peer 97d16299dd36478e89644f3b7d82ea1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.340029 9721 raft_consensus.cc:1261] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Refusing update from remote peer 97d16299dd36478e89644f3b7d82ea1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.340222 10122 consensus_queue.cc:1035] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.340307 10105 consensus_queue.cc:1035] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.357286 9562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.357385 9562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.357998 9562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Sending DeleteTablet for 3 replicas of tablet dda75cb559f04f3194e3b17695ee0902
I20260928 14:58:26.358224 9684 tablet_service.cc:1514] Processing DeleteTablet for tablet dda75cb559f04f3194e3b17695ee0902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260928 14:58:26.358248 9975 tablet_service.cc:1514] Processing DeleteTablet for tablet dda75cb559f04f3194e3b17695ee0902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59342
I20260928 14:58:26.358317 10099 tablet_replica.cc:330] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: stopping tablet replica
I20260928 14:58:26.358366 10098 tablet_replica.cc:330] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: stopping tablet replica
I20260928 14:58:26.358364 10099 raft_consensus.cc:2229] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.358405 10099 raft_consensus.cc:2258] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.358388 9840 tablet_service.cc:1514] Processing DeleteTablet for tablet dda75cb559f04f3194e3b17695ee0902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:51482
I20260928 14:58:26.358421 10098 raft_consensus.cc:2229] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.358458 10098 raft_consensus.cc:2258] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.358464 10100 tablet_replica.cc:330] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: stopping tablet replica
I20260928 14:58:26.358508 10100 raft_consensus.cc:2229] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:26.358589 10100 raft_consensus.cc:2258] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.358718 10099 ts_tablet_manager.cc:1905] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.358767 10098 ts_tablet_manager.cc:1905] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.358892 10100 ts_tablet_manager.cc:1905] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.360589 10098 ts_tablet_manager.cc:1918] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:26.360600 10099 ts_tablet_manager.cc:1918] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:26.360647 10098 log.cc:1196] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/dda75cb559f04f3194e3b17695ee0902
I20260928 14:58:26.360659 10099 log.cc:1196] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/dda75cb559f04f3194e3b17695ee0902
I20260928 14:58:26.360870 10098 ts_tablet_manager.cc:1939] T dda75cb559f04f3194e3b17695ee0902 P b1f222e0c392457ba13d4371e0d41568: Deleting consensus metadata
I20260928 14:58:26.360886 10099 ts_tablet_manager.cc:1939] T dda75cb559f04f3194e3b17695ee0902 P d228e9c255824229b62ff13a94e83f21: Deleting consensus metadata
I20260928 14:58:26.361176 9548 catalog_manager.cc:4841] TS d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143): tablet dda75cb559f04f3194e3b17695ee0902 (table test-table [id=103582e2d0c4422c8f53c971e299040b]) successfully deleted
I20260928 14:58:26.361377 9551 catalog_manager.cc:4841] TS b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911): tablet dda75cb559f04f3194e3b17695ee0902 (table test-table [id=103582e2d0c4422c8f53c971e299040b]) successfully deleted
I20260928 14:58:26.363524 10100 ts_tablet_manager.cc:1918] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:26.363588 10100 log.cc:1196] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/dda75cb559f04f3194e3b17695ee0902
I20260928 14:58:26.363790 10100 ts_tablet_manager.cc:1939] T dda75cb559f04f3194e3b17695ee0902 P 97d16299dd36478e89644f3b7d82ea1a: Deleting consensus metadata
I20260928 14:58:26.364118 9549 catalog_manager.cc:4841] TS 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735): tablet dda75cb559f04f3194e3b17695ee0902 (table test-table [id=103582e2d0c4422c8f53c971e299040b]) successfully deleted
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:58:26.386425 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:26.380496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.1.252.254:45925)
0928 14:58:26.380553 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.380556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.380669 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:58:26.380740 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.380740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.380750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.380804 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.381510 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.382007 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.383101 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.383284 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.383351 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.386355 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41166: BlockingRecv error: recv got EOF from 127.0.0.1:41166 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[9519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM
I20260928 14:58:26.394915 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.387364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.1.252.254:45925)
0928 14:58:26.387408 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.387410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.387494 (+ 84us) server_negotiation.cc:407] Connection header received
0928 14:58:26.387534 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.387535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.387544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.387599 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:26.388093 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.388857 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.389549 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.389820 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.389893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.392092 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:26.392094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:26.392094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:26.392102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:26.393643 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.393999 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.394001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.394001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.394067 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:26.394331 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:26.394332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:26.394333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:26.394472 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:26.394579 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.394745 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.394822 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:58:26.396428 9562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:26.396615 9562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:26.398175 9684 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3bc5e0aecb476b83ad238f8f773c59 (DEFAULT_TABLE table=test-table [id=599321fdc5d14e85b002caf50e2dbac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.398205 9840 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3bc5e0aecb476b83ad238f8f773c59 (DEFAULT_TABLE table=test-table [id=599321fdc5d14e85b002caf50e2dbac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.398214 9975 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3bc5e0aecb476b83ad238f8f773c59 (DEFAULT_TABLE table=test-table [id=599321fdc5d14e85b002caf50e2dbac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.398341 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bc5e0aecb476b83ad238f8f773c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.398355 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bc5e0aecb476b83ad238f8f773c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.398372 9840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bc5e0aecb476b83ad238f8f773c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.399755 10075 tablet_bootstrap.cc:492] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Bootstrap starting.
I20260928 14:58:26.399809 10079 tablet_bootstrap.cc:492] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Bootstrap starting.
I20260928 14:58:26.400151 10075 tablet_bootstrap.cc:654] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.400372 10079 tablet_bootstrap.cc:654] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.400450 10078 tablet_bootstrap.cc:492] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Bootstrap starting.
I20260928 14:58:26.400799 10078 tablet_bootstrap.cc:654] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.401065 10079 tablet_bootstrap.cc:492] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: No bootstrap required, opened a new log
I20260928 14:58:26.401116 10079 ts_tablet_manager.cc:1397] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:26.401260 10079 raft_consensus.cc:348] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.401345 10079 raft_consensus.cc:374] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.401365 10079 raft_consensus.cc:729] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d228e9c255824229b62ff13a94e83f21, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.401415 10079 consensus_queue.cc:260] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.401516 10079 ts_tablet_manager.cc:1428] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:26.401574 10075 tablet_bootstrap.cc:492] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: No bootstrap required, opened a new log
I20260928 14:58:26.401613 10075 ts_tablet_manager.cc:1397] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 14:58:26.401718 10075 raft_consensus.cc:348] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.401770 10075 raft_consensus.cc:374] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.401787 10075 raft_consensus.cc:729] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.401840 10075 consensus_queue.cc:260] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.401916 10075 ts_tablet_manager.cc:1428] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.402473 10078 tablet_bootstrap.cc:492] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: No bootstrap required, opened a new log
I20260928 14:58:26.402529 10078 ts_tablet_manager.cc:1397] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:58:26.402693 10078 raft_consensus.cc:348] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.402767 10078 raft_consensus.cc:374] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.402789 10078 raft_consensus.cc:729] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.402837 10078 consensus_queue.cc:260] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.402927 10078 ts_tablet_manager.cc:1428] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:26.430693 10095 raft_consensus.cc:482] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:26.430801 10095 raft_consensus.cc:504] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.431020 10095 leader_election.cc:288] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735)
I20260928 14:58:26.431209 9720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bc5e0aecb476b83ad238f8f773c59" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21" is_pre_election: true
I20260928 14:58:26.431262 9861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bc5e0aecb476b83ad238f8f773c59" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d16299dd36478e89644f3b7d82ea1a" is_pre_election: true
I20260928 14:58:26.431318 9720 raft_consensus.cc:2454] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 0.
I20260928 14:58:26.431335 9861 raft_consensus.cc:2454] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 0.
I20260928 14:58:26.431490 9932 leader_election.cc:302] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f222e0c392457ba13d4371e0d41568, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.431579 10095 raft_consensus.cc:2781] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:26.431614 10095 raft_consensus.cc:482] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:26.431643 10095 raft_consensus.cc:3037] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.432440 10095 raft_consensus.cc:504] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.432576 10095 leader_election.cc:288] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 election: Requested vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735)
I20260928 14:58:26.432711 9720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bc5e0aecb476b83ad238f8f773c59" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21"
I20260928 14:58:26.432789 9720 raft_consensus.cc:3037] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.432761 9861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bc5e0aecb476b83ad238f8f773c59" candidate_uuid: "b1f222e0c392457ba13d4371e0d41568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d16299dd36478e89644f3b7d82ea1a"
I20260928 14:58:26.432824 9861 raft_consensus.cc:3037] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.433562 9861 raft_consensus.cc:2454] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 1.
I20260928 14:58:26.433636 9720 raft_consensus.cc:2454] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f222e0c392457ba13d4371e0d41568 in term 1.
I20260928 14:58:26.433784 9932 leader_election.cc:302] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f222e0c392457ba13d4371e0d41568, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.433861 10095 raft_consensus.cc:2781] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:26.433917 10095 raft_consensus.cc:686] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 1 LEADER]: Becoming Leader. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Running, Role: LEADER
I20260928 14:58:26.433972 10095 consensus_queue.cc:237] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.434554 9562 catalog_manager.cc:5495] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1f222e0c392457ba13d4371e0d41568 (127.1.252.195). New cstate: current_term: 1 leader_uuid: "b1f222e0c392457ba13d4371e0d41568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } health_report { overall_health: HEALTHY } } }
I20260928 14:58:26.440780 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.438396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.1.252.195:35911)
0928 14:58:26.438461 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.438463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.438473 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:26.438533 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.438535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.438545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.438600 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:26.438745 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.439221 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.440091 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.440296 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.440371 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.440537 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.440576 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.440634 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.440687 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.440708 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:58:26.441463 9861 raft_consensus.cc:1261] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Refusing update from remote peer b1f222e0c392457ba13d4371e0d41568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.441489 9720 raft_consensus.cc:1261] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Refusing update from remote peer b1f222e0c392457ba13d4371e0d41568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.441695 10095 consensus_queue.cc:1035] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.441812 10085 consensus_queue.cc:1035] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.444659 9562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.444746 9562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.445413 9562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Sending DeleteTablet for 3 replicas of tablet 1d3bc5e0aecb476b83ad238f8f773c59
I20260928 14:58:26.445621 9684 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3bc5e0aecb476b83ad238f8f773c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260928 14:58:26.445668 9840 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3bc5e0aecb476b83ad238f8f773c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:51482
I20260928 14:58:26.445709 10099 tablet_replica.cc:330] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: stopping tablet replica
I20260928 14:58:26.445703 9975 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3bc5e0aecb476b83ad238f8f773c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59342
I20260928 14:58:26.445751 10099 raft_consensus.cc:2229] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.445749 10100 tablet_replica.cc:330] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: stopping tablet replica
I20260928 14:58:26.445791 10099 raft_consensus.cc:2258] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.445797 10100 raft_consensus.cc:2229] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.445837 10100 raft_consensus.cc:2258] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.445858 10098 tablet_replica.cc:330] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: stopping tablet replica
I20260928 14:58:26.445911 10098 raft_consensus.cc:2229] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:26.445955 10098 raft_consensus.cc:2258] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.446130 10100 ts_tablet_manager.cc:1905] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.446257 10098 ts_tablet_manager.cc:1905] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.446794 10099 ts_tablet_manager.cc:1905] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.447640 10098 ts_tablet_manager.cc:1918] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:26.447696 10098 log.cc:1196] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/1d3bc5e0aecb476b83ad238f8f773c59
I20260928 14:58:26.447659 9562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:26.447809 9562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:26.447942 10098 ts_tablet_manager.cc:1939] T 1d3bc5e0aecb476b83ad238f8f773c59 P b1f222e0c392457ba13d4371e0d41568: Deleting consensus metadata
I20260928 14:58:26.448158 10099 ts_tablet_manager.cc:1918] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:26.448204 10099 log.cc:1196] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/1d3bc5e0aecb476b83ad238f8f773c59
I20260928 14:58:26.448374 9551 catalog_manager.cc:4841] TS b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911): tablet 1d3bc5e0aecb476b83ad238f8f773c59 (table test-table [id=599321fdc5d14e85b002caf50e2dbac4]) successfully deleted
I20260928 14:58:26.448417 10099 ts_tablet_manager.cc:1939] T 1d3bc5e0aecb476b83ad238f8f773c59 P d228e9c255824229b62ff13a94e83f21: Deleting consensus metadata
I20260928 14:58:26.448915 9548 catalog_manager.cc:4841] TS d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143): tablet 1d3bc5e0aecb476b83ad238f8f773c59 (table test-table [id=599321fdc5d14e85b002caf50e2dbac4]) successfully deleted
I20260928 14:58:26.448952 10100 ts_tablet_manager.cc:1918] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:26.449003 10100 log.cc:1196] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/1d3bc5e0aecb476b83ad238f8f773c59
I20260928 14:58:26.449201 10100 ts_tablet_manager.cc:1939] T 1d3bc5e0aecb476b83ad238f8f773c59 P 97d16299dd36478e89644f3b7d82ea1a: Deleting consensus metadata
I20260928 14:58:26.449522 9549 catalog_manager.cc:4841] TS 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194:37735): tablet 1d3bc5e0aecb476b83ad238f8f773c59 (table test-table [id=599321fdc5d14e85b002caf50e2dbac4]) successfully deleted
I20260928 14:58:26.449793 9840 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfd1b9db30842c48789c40398c5e9d6 (DEFAULT_TABLE table=test-table [id=72f1974ae73e407a9c071f07f7ac8a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.449810 9684 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfd1b9db30842c48789c40398c5e9d6 (DEFAULT_TABLE table=test-table [id=72f1974ae73e407a9c071f07f7ac8a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.449904 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfd1b9db30842c48789c40398c5e9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.449904 9840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfd1b9db30842c48789c40398c5e9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.451050 10079 tablet_bootstrap.cc:492] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: Bootstrap starting.
I20260928 14:58:26.451059 9975 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfd1b9db30842c48789c40398c5e9d6 (DEFAULT_TABLE table=test-table [id=72f1974ae73e407a9c071f07f7ac8a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:26.451153 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfd1b9db30842c48789c40398c5e9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:26.451659 10079 tablet_bootstrap.cc:654] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.452003 10078 tablet_bootstrap.cc:492] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: Bootstrap starting.
I20260928 14:58:26.452127 10075 tablet_bootstrap.cc:492] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Bootstrap starting.
I20260928 14:58:26.452353 10078 tablet_bootstrap.cc:654] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.452450 10075 tablet_bootstrap.cc:654] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:26.452893 10079 tablet_bootstrap.cc:492] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: No bootstrap required, opened a new log
I20260928 14:58:26.452945 10079 ts_tablet_manager.cc:1397] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:26.453094 10079 raft_consensus.cc:348] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.453167 10079 raft_consensus.cc:374] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.453195 10079 raft_consensus.cc:729] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d228e9c255824229b62ff13a94e83f21, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.453239 10075 tablet_bootstrap.cc:492] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: No bootstrap required, opened a new log
I20260928 14:58:26.453235 10079 consensus_queue.cc:260] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.453289 10075 ts_tablet_manager.cc:1397] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:58:26.453331 10079 ts_tablet_manager.cc:1428] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.453390 10075 raft_consensus.cc:348] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.453454 10075 raft_consensus.cc:374] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.453481 10075 raft_consensus.cc:729] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f222e0c392457ba13d4371e0d41568, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.453526 10075 consensus_queue.cc:260] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.453619 10075 ts_tablet_manager.cc:1428] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.453718 10078 tablet_bootstrap.cc:492] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: No bootstrap required, opened a new log
I20260928 14:58:26.453770 10078 ts_tablet_manager.cc:1397] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:58:26.453900 10078 raft_consensus.cc:348] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.453967 10078 raft_consensus.cc:374] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:26.453989 10078 raft_consensus.cc:729] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Initialized, Role: FOLLOWER
I20260928 14:58:26.454027 10078 consensus_queue.cc:260] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.454130 10078 ts_tablet_manager.cc:1428] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:26.497455 10105 raft_consensus.cc:482] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:26.497582 10105 raft_consensus.cc:504] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.497793 10105 leader_election.cc:288] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
I20260928 14:58:26.497962 9720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfd1b9db30842c48789c40398c5e9d6" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21" is_pre_election: true
I20260928 14:58:26.497975 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfd1b9db30842c48789c40398c5e9d6" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f222e0c392457ba13d4371e0d41568" is_pre_election: true
I20260928 14:58:26.498077 9720 raft_consensus.cc:2454] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 0.
I20260928 14:58:26.498095 10001 raft_consensus.cc:2454] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 0.
I20260928 14:58:26.498256 9792 leader_election.cc:302] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.498374 10105 raft_consensus.cc:2781] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:26.498436 10105 raft_consensus.cc:482] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:26.498469 10105 raft_consensus.cc:3037] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.499368 10105 raft_consensus.cc:504] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.499507 10105 leader_election.cc:288] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 election: Requested vote from peers d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143), b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911)
I20260928 14:58:26.499646 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfd1b9db30842c48789c40398c5e9d6" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f222e0c392457ba13d4371e0d41568"
I20260928 14:58:26.499719 10001 raft_consensus.cc:3037] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.499665 9720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfd1b9db30842c48789c40398c5e9d6" candidate_uuid: "97d16299dd36478e89644f3b7d82ea1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d228e9c255824229b62ff13a94e83f21"
I20260928 14:58:26.499766 9720 raft_consensus.cc:3037] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:26.500470 9720 raft_consensus.cc:2454] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 1.
I20260928 14:58:26.500578 10001 raft_consensus.cc:2454] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d16299dd36478e89644f3b7d82ea1a in term 1.
I20260928 14:58:26.500641 9792 leader_election.cc:302] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d16299dd36478e89644f3b7d82ea1a, d228e9c255824229b62ff13a94e83f21; no voters:
I20260928 14:58:26.500710 10105 raft_consensus.cc:2781] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:26.500756 10105 raft_consensus.cc:686] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 1 LEADER]: Becoming Leader. State: Replica: 97d16299dd36478e89644f3b7d82ea1a, State: Running, Role: LEADER
I20260928 14:58:26.500801 10105 consensus_queue.cc:237] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } }
I20260928 14:58:26.501348 9562 catalog_manager.cc:5495] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97d16299dd36478e89644f3b7d82ea1a (127.1.252.194). New cstate: current_term: 1 leader_uuid: "97d16299dd36478e89644f3b7d82ea1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d16299dd36478e89644f3b7d82ea1a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:26.517063 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.514785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.1.252.194:37735)
0928 14:58:26.514840 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:26.514842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:26.514849 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:58:26.514875 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:26.514876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:26.514884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:26.514945 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:26.515069 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.515529 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.516340 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:26.516534 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:26.516619 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:26.516776 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:26.516814 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:26.516907 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:26.516970 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 14:58:26.516989 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:58:26.519219 9720 raft_consensus.cc:1261] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Refusing update from remote peer 97d16299dd36478e89644f3b7d82ea1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.519411 10105 consensus_queue.cc:1035] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d228e9c255824229b62ff13a94e83f21" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.519462 10001 raft_consensus.cc:1261] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Refusing update from remote peer 97d16299dd36478e89644f3b7d82ea1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:26.519630 10105 consensus_queue.cc:1035] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f222e0c392457ba13d4371e0d41568" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:26.529320 9562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.529412 9562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:26.530030 9562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 967e29b9571a4c3287f028aa7a0905e6: Sending DeleteTablet for 3 replicas of tablet 7cfd1b9db30842c48789c40398c5e9d6
I20260928 14:58:26.530274 9684 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cfd1b9db30842c48789c40398c5e9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260928 14:58:26.530370 10099 tablet_replica.cc:330] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: stopping tablet replica
I20260928 14:58:26.530368 9975 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cfd1b9db30842c48789c40398c5e9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59342
I20260928 14:58:26.530423 10099 raft_consensus.cc:2229] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.530457 10099 raft_consensus.cc:2258] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.530457 10098 tablet_replica.cc:330] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: stopping tablet replica
I20260928 14:58:26.530512 10098 raft_consensus.cc:2229] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:26.530550 10098 raft_consensus.cc:2258] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.530629 9840 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cfd1b9db30842c48789c40398c5e9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:26 UTC) from {username='oryx', principal='oryx/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:51482
I20260928 14:58:26.530704 10100 tablet_replica.cc:330] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: stopping tablet replica
I20260928 14:58:26.530736 10100 raft_consensus.cc:2229] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:26.530776 10100 raft_consensus.cc:2258] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:26.530953 10099 ts_tablet_manager.cc:1905] T 7cfd1b9db30842c48789c40398c5e9d6 P d228e9c255824229b62ff13a94e83f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.531002 10098 ts_tablet_manager.cc:1905] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:26.531203 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 9636
I20260928 14:58:26.532791 10098 ts_tablet_manager.cc:1918] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:58:26.532842 10098 log.cc:1196] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/7cfd1b9db30842c48789c40398c5e9d6
I20260928 14:58:26.533046 10098 ts_tablet_manager.cc:1939] T 7cfd1b9db30842c48789c40398c5e9d6 P b1f222e0c392457ba13d4371e0d41568: Deleting consensus metadata
I20260928 14:58:26.533504 9551 catalog_manager.cc:4841] TS b1f222e0c392457ba13d4371e0d41568 (127.1.252.195:35911): tablet 7cfd1b9db30842c48789c40398c5e9d6 (table test-table [id=72f1974ae73e407a9c071f07f7ac8a8b]) successfully deleted
W20260928 14:58:26.537581 9548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:43143 (error 108)
I20260928 14:58:26.537873 10100 ts_tablet_manager.cc:1905] T 7cfd1b9db30842c48789c40398c5e9d6 P 97d16299dd36478e89644f3b7d82ea1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:58:26.538292 10069 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:26.537988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:43143 (local address 127.0.0.1:35224)
0928 14:58:26.538032 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.538042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:26.538089 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:26.538238 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:43143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 14:58:26.538411 9548 catalog_manager.cc:4584] TS d228e9c255824229b62ff13a94e83f21 (127.1.252.193:43143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cfd1b9db30842c48789c40398c5e9d6: Network error: Client connection negotiation failed: client connection to 127.1.252.193:43143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:58:26.538666 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 9776
I20260928 14:58:26.544533 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:26.544389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:37735 (local address 127.0.0.1:51522)
0928 14:58:26.544449 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:26.544504 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.194:37735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:58:26.544559 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 9916
I20260928 14:58:26.549831 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 9532
2026-09-28T14:58:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2342 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[10157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[10157](info): set up 2 sockets
Sep 28 14:58:26 dist-test-slave-sf70 krb5kdc[10157](info): commencing operation
krb5kdc: starting...
W20260928 14:58:27.606130 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:27 dist-test-slave-sf70 krb5kdc[10157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:27.723773 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33973
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:33973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:27.804270 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:27.804455 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:27.804476 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:27.805521 10170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:27.805553 10170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:27.805567 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:27.805580 10170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:27.805593 10170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:27.806962 10170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:33973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:27.807132 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:27.807283 10170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:27.811794 10175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:27.811828 10178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:27.811800 10176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:27.811954 10170 server_base.cc:1034] running on GCE node
I20260928 14:58:27.812378 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:27.812645 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:27.813809 10170 hybrid_clock.cc:648] HybridClock initialized: now 1790607507813793 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:58:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1270 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:27 dist-test-slave-sf70 krb5kdc[10183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:27 dist-test-slave-sf70 krb5kdc[10183](info): set up 2 sockets
Sep 28 14:58:27 dist-test-slave-sf70 krb5kdc[10183](info): commencing operation
krb5kdc: starting...
W20260928 14:58:28.866236 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:28 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:28.988062 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42277
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:29.071148 10196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:29.071342 10196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:29.071364 10196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:29.072445 10196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:29.072479 10196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:29.072494 10196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:29.072505 10196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:29.072517 10196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:29.073880 10196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:29.074069 10196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:29.074222 10196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:29.078899 10202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.078927 10201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.078982 10204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.079375 10196 server_base.cc:1034] running on GCE node
I20260928 14:58:29.079568 10196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:29.079802 10196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:29.080947 10196 hybrid_clock.cc:648] HybridClock initialized: now 1790607509080933 us; error 29 us; skew 500 ppm
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:29.083456 10196 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:29.083813 10196 webserver.cc:460] Webserver started at http://127.1.252.254:43813/ using document root <none> and password file <none>
I20260928 14:58:29.084062 10196 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:29.084132 10196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:29.084290 10196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:29.085285 10196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "588960d9fc0a4ea6bbe99b83594c8b0f"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "656d7ee61901a2e5fa435077de10609c"
server_key_iv: "2a05a992145509a454da043115349464"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.085644 10196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "588960d9fc0a4ea6bbe99b83594c8b0f"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "656d7ee61901a2e5fa435077de10609c"
server_key_iv: "2a05a992145509a454da043115349464"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.086963 10196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:29.087668 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.087867 10196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:29.087987 10196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "588960d9fc0a4ea6bbe99b83594c8b0f"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "656d7ee61901a2e5fa435077de10609c"
server_key_iv: "2a05a992145509a454da043115349464"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.088068 10196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:29.111992 10196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:29.114418 10196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:29.114586 10196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:29.118455 10196 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:42277
I20260928 14:58:29.118527 10263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:42277 every 8 connection(s)
I20260928 14:58:29.118871 10196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:29.119689 10264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:29.121948 10264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Bootstrap starting.
I20260928 14:58:29.122368 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10196
I20260928 14:58:29.122471 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:29.122613 10264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:29.122677 2035 external_mini_cluster.cc:1428] Setting key 4f4754cc332b88cfd0697a5df43a4ab6
I20260928 14:58:29.122987 10264 log.cc:824] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:29.123710 10264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: No bootstrap required, opened a new log
I20260928 14:58:29.125579 10264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.125732 10264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:29.125785 10264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588960d9fc0a4ea6bbe99b83594c8b0f, State: Initialized, Role: FOLLOWER
I20260928 14:58:29.125916 10264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.125977 10264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:29.125998 10264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:29.126046 10264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:29.126621 10264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.126737 10264 leader_election.cc:302] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588960d9fc0a4ea6bbe99b83594c8b0f; no voters:
I20260928 14:58:29.126921 10264 leader_election.cc:288] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:29.126959 10269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:29.127126 10269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 LEADER]: Becoming Leader. State: Replica: 588960d9fc0a4ea6bbe99b83594c8b0f, State: Running, Role: LEADER
I20260928 14:58:29.127192 10264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:29.127270 10269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.127600 10269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588960d9fc0a4ea6bbe99b83594c8b0f. Latest consensus state: current_term: 1 leader_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } } }
I20260928 14:58:29.127710 10269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:29.127879 10271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } } }
I20260928 14:58:29.128026 10271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:29.128089 10275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:29.128650 10275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:29.130195 10275 catalog_manager.cc:1347] Generated new cluster ID: 482ece474f324c078703637e0303f50c
I20260928 14:58:29.130252 10275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:29.133875 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.123254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.1.252.254:42277)
0928 14:58:29.123438 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.123448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.123480 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:58:29.123981 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.123987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.124034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.124165 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.124800 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.125674 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.126595 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.127199 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.127612 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.130164 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:29.130168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:29.130168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:29.130179 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:29.132110 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.132519 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.132522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.132522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.132627 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.132833 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.132834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.132834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.133131 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:29.133285 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:29.133418 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 14:58:29.133662 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260928 14:58:29.134045 10275 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:29.134632 10275 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:29.148319 10275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Generated new TSK 0
I20260928 14:58:29.148502 10275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:29.179625 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42277
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:29.260787 10290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:29.261040 10290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:29.261081 10290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:29.262245 10290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:29.262316 10290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:29.262358 10290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:29.263788 10290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:29.264081 10290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:29.264309 10290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:29.269285 10296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.269307 10298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.269299 10295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.269384 10290 server_base.cc:1034] running on GCE node
I20260928 14:58:29.269726 10290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:29.269980 10290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:29.271126 10290 hybrid_clock.cc:648] HybridClock initialized: now 1790607509271102 us; error 40 us; skew 500 ppm
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:29.273715 10290 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:29.274159 10290 webserver.cc:460] Webserver started at http://127.1.252.193:32969/ using document root <none> and password file <none>
I20260928 14:58:29.274411 10290 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:29.274482 10290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:29.274626 10290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:29.275641 10290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "367e2affa15b43e894bea0adee3f96e9"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "1f53c55b7ba52b1f398714f838e73fab"
server_key_iv: "bc3eebeaf9381f840f7ac12885eea2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.276074 10290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "367e2affa15b43e894bea0adee3f96e9"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "1f53c55b7ba52b1f398714f838e73fab"
server_key_iv: "bc3eebeaf9381f840f7ac12885eea2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.277393 10290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:29.278118 10305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.278312 10290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:29.278420 10290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "367e2affa15b43e894bea0adee3f96e9"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "1f53c55b7ba52b1f398714f838e73fab"
server_key_iv: "bc3eebeaf9381f840f7ac12885eea2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.278504 10290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:29.305140 10290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:29.307132 10290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:29.307297 10290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:29.307549 10290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:29.307858 10290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:29.307947 10290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.307989 10290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:29.308036 10290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.313931 10290 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:32835
I20260928 14:58:29.313980 10418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:32835 every 8 connection(s)
I20260928 14:58:29.314280 10290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:29.314788 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10290
I20260928 14:58:29.314918 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:29.315143 2035 external_mini_cluster.cc:1428] Setting key 3579ef71518f013513ad3ed212cd1581
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 14:58:29.324915 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.314994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:54447 (local address 127.1.252.254:42277)
0928 14:58:29.315071 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.315074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.315315 (+ 241us) server_negotiation.cc:407] Connection header received
0928 14:58:29.315992 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.315993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.316005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.316063 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.316962 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.317745 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.318720 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.319070 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.319146 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.321734 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:29.321735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:29.321736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:29.321743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:29.323395 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.323876 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.323877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.323877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.323979 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.324251 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.324252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.324253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.324369 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:29.324442 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:29.324739 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:58:29.324826 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:58:29.325073 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.314985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:54447)
0928 14:58:29.315173 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:29.315227 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:29.315843 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:29.316123 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:29.316126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:29.316672 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:29.316855 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.316859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.317846 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.317847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.318636 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.318639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.319229 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.319230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.319347 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.319993 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:29.320002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:29.321638 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:29.323490 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:29.323497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:29.323499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:29.323710 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:29.324052 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:29.324053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:29.324054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:29.324160 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:29.324448 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:29.324455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:29.324578 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 14:58:29.324692 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:58:29.324848 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260928 14:58:29.325572 10419 heartbeater.cc:347] Connected to a master server at 127.1.252.254:42277
I20260928 14:58:29.325660 10419 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:29.325841 10419 heartbeater.cc:511] Master 127.1.252.254:42277 requested a full tablet report, sending...
I20260928 14:58:29.326331 10228 ts_manager.cc:194] Registered new tserver with Master: 367e2affa15b43e894bea0adee3f96e9 (127.1.252.193:32835)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
I20260928 14:58:29.327010 10228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:54447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:29.343688 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42277
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:29.423918 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:29.424110 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:29.424134 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:29.425298 10424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:29.425340 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:29.425398 10424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:29.426857 10424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:29.427088 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:29.427309 10424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:29.432401 10432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.432381 10429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.432382 10430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.432529 10424 server_base.cc:1034] running on GCE node
I20260928 14:58:29.432775 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:29.432977 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:29.434111 10424 hybrid_clock.cc:648] HybridClock initialized: now 1790607509434102 us; error 32 us; skew 500 ppm
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:29.436604 10424 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:29.437021 10424 webserver.cc:460] Webserver started at http://127.1.252.194:44703/ using document root <none> and password file <none>
I20260928 14:58:29.437245 10424 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:29.437291 10424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:29.437462 10424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:29.438493 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "4ef6030c87b144aaae2b84caea28c6ba"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "e19d2a454389f26ce836ae0119901e26"
server_key_iv: "e08b8bc29f4a45101d6195b40f7f4433"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.438854 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "4ef6030c87b144aaae2b84caea28c6ba"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "e19d2a454389f26ce836ae0119901e26"
server_key_iv: "e08b8bc29f4a45101d6195b40f7f4433"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.440212 10424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:58:29.440892 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.441054 10424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:29.441152 10424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "4ef6030c87b144aaae2b84caea28c6ba"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "e19d2a454389f26ce836ae0119901e26"
server_key_iv: "e08b8bc29f4a45101d6195b40f7f4433"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.441233 10424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:29.452667 10424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:29.455014 10424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:29.455178 10424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:29.455417 10424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:29.455732 10424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:29.455788 10424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.455832 10424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:29.455856 10424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.461771 10424 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:43651
I20260928 14:58:29.461815 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:43651 every 8 connection(s)
I20260928 14:58:29.462121 10424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:29.468106 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10424
I20260928 14:58:29.468241 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:29.468436 2035 external_mini_cluster.cc:1428] Setting key cbb7006f69a3d846c21c842b33ba340c
I20260928 14:58:29.472569 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.462832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50073 (local address 127.1.252.254:42277)
0928 14:58:29.462895 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.462897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.463092 (+ 195us) server_negotiation.cc:407] Connection header received
0928 14:58:29.463624 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.463625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.463638 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.463681 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.464690 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.465192 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.466295 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.466495 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.466602 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.468900 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:29.468901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:29.468901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:29.468907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:29.470587 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.471192 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.471193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.471194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.471287 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.471617 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.471618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.471619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.471789 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:29.471867 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:29.472241 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 14:58:29.472391 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:58:29.472630 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.462812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:50073)
0928 14:58:29.462978 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:29.463013 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:29.463517 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:29.463746 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:29.463751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:29.464379 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:29.464571 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.464575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.465289 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.465290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.466195 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.466198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.466609 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.466610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.466737 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.467195 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:29.467203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:29.468815 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:29.470707 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:29.470736 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:29.470739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:29.471081 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:29.471375 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:29.471376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:29.471376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:29.471519 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:29.471857 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:29.471863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:29.472063 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 14:58:29.472179 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:58:29.472356 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 14:58:29.476554 10553 heartbeater.cc:347] Connected to a master server at 127.1.252.254:42277
I20260928 14:58:29.476670 10553 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:29.476863 10553 heartbeater.cc:511] Master 127.1.252.254:42277 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
I20260928 14:58:29.477300 10228 ts_manager.cc:194] Registered new tserver with Master: 4ef6030c87b144aaae2b84caea28c6ba (127.1.252.194:43651)
I20260928 14:58:29.477798 10228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:50073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:29.496114 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42277
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:29.575145 10558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:29.575373 10558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:29.575412 10558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:29.576609 10558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:29.576675 10558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:29.576721 10558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:29.578146 10558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:29.578405 10558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:29.578630 10558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:29.583776 10563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.583777 10566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.583937 10564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.583832 10558 server_base.cc:1034] running on GCE node
I20260928 14:58:29.584177 10558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:29.584388 10558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:29.585530 10558 hybrid_clock.cc:648] HybridClock initialized: now 1790607509585515 us; error 30 us; skew 500 ppm
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:29.588017 10558 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:29.588428 10558 webserver.cc:460] Webserver started at http://127.1.252.195:40623/ using document root <none> and password file <none>
I20260928 14:58:29.588670 10558 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:29.588737 10558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:29.588871 10558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:29.589926 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "705cd72336364eccbff4b701ac81e071"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "9898d242c9428e43037610956b5c5549"
server_key_iv: "0c762c806e44d8fb63e9caadde1eb5a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.590401 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "705cd72336364eccbff4b701ac81e071"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "9898d242c9428e43037610956b5c5549"
server_key_iv: "0c762c806e44d8fb63e9caadde1eb5a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.591753 10558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:29.592551 10573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.592717 10558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:29.592813 10558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "705cd72336364eccbff4b701ac81e071"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "9898d242c9428e43037610956b5c5549"
server_key_iv: "0c762c806e44d8fb63e9caadde1eb5a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.592901 10558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:29.605773 10558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:29.608054 10558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:29.608206 10558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:29.608428 10558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:29.608724 10558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:29.608777 10558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.608815 10558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:29.608842 10558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.614440 10558 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:38223
I20260928 14:58:29.614508 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:38223 every 8 connection(s)
I20260928 14:58:29.614823 10558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:29.620571 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10558
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:29.620687 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:29.620862 2035 external_mini_cluster.cc:1428] Setting key b2b2f868e368a469295c3abf41767f63
I20260928 14:58:29.624011 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.615540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:39815 (local address 127.1.252.254:42277)
0928 14:58:29.615608 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.615610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.615798 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:58:29.616330 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.616331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.616342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.616400 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.617275 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.617759 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.618668 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.618847 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.618904 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.621216 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:29.621217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:29.621218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:29.621225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:29.622622 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.623039 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.623040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.623041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.623102 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:29.623349 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:29.623350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:29.623351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:29.623470 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:29.623546 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:29.623810 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 14:58:29.623925 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:58:29.624179 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.615509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:39815)
0928 14:58:29.615684 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:29.615717 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:29.616241 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:29.616449 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:29.616452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:29.617011 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:29.617195 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.617199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.617835 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.617835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.618590 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.618592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.618940 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.618941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.619035 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.619505 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:29.619520 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:29.621117 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:29.622725 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:29.622732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:29.622734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:29.622948 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:29.623173 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:29.623174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:29.623174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:29.623282 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:29.623550 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:29.623556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:29.623685 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 14:58:29.623789 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:58:29.623957 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260928 14:58:29.624599 10687 heartbeater.cc:347] Connected to a master server at 127.1.252.254:42277
I20260928 14:58:29.624704 10687 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:29.624872 10687 heartbeater.cc:511] Master 127.1.252.254:42277 requested a full tablet report, sending...
I20260928 14:58:29.625252 10228 ts_manager.cc:194] Registered new tserver with Master: 705cd72336364eccbff4b701ac81e071 (127.1.252.195:38223)
I20260928 14:58:29.625622 10228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:39815
I20260928 14:58:29.632470 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab.
Entry for principal oryx/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab.
WARNING: no policy specified for oryx/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab.
Entry for principal oryx/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab.
WARNING: no policy specified for oryx/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab.
Entry for principal oryx/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab.
I20260928 14:58:29.706863 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10290
I20260928 14:58:29.711401 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10424
I20260928 14:58:29.715953 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10558
I20260928 14:58:29.720059 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10196
I20260928 14:58:29.724972 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42277
--webserver_interface=127.1.252.254
--webserver_port=43813
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:29.804875 10696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:29.805065 10696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:29.805086 10696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:29.806131 10696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:29.806167 10696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:29.806181 10696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:29.806195 10696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:29.806205 10696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:29.807611 10696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:29.807782 10696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:29.808012 10696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:29.812567 10702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.812578 10701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.812705 10696 server_base.cc:1034] running on GCE node
W20260928 14:58:29.812566 10704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.813083 10696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:29.813313 10696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:29.814474 10696 hybrid_clock.cc:648] HybridClock initialized: now 1790607509814460 us; error 43 us; skew 500 ppm
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:29.816991 10696 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:29.817432 10696 webserver.cc:460] Webserver started at http://127.1.252.254:43813/ using document root <none> and password file <none>
I20260928 14:58:29.817662 10696 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:29.817744 10696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:29.819041 10696 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:29.819653 10711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.819808 10696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:29.819932 10696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "588960d9fc0a4ea6bbe99b83594c8b0f"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "656d7ee61901a2e5fa435077de10609c"
server_key_iv: "2a05a992145509a454da043115349464"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.820240 10696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:29.829787 10696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:29.831619 10696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:29.831769 10696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:29.835650 10696 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:42277
I20260928 14:58:29.835680 10763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:42277 every 8 connection(s)
I20260928 14:58:29.836095 10696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:29.837445 10764 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:58:29.838001 10764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Bootstrap starting.
I20260928 14:58:29.839289 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10696
I20260928 14:58:29.839584 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:32835
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:42277
--webserver_port=32969
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:58:29.846264 10764 log.cc:824] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:29.847791 10764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:58:29.848114 10764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Bootstrap complete.
I20260928 14:58:29.850323 10764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.850543 10764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588960d9fc0a4ea6bbe99b83594c8b0f, State: Initialized, Role: FOLLOWER
I20260928 14:58:29.850708 10764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.850778 10764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:29.850838 10764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:29.850893 10764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:58:29.851856 10764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.852005 10764 leader_election.cc:302] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588960d9fc0a4ea6bbe99b83594c8b0f; no voters:
I20260928 14:58:29.852190 10764 leader_election.cc:288] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:58:29.852221 10768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:58:29.852340 10768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [term 2 LEADER]: Becoming Leader. State: Replica: 588960d9fc0a4ea6bbe99b83594c8b0f, State: Running, Role: LEADER
I20260928 14:58:29.852448 10764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:29.852430 10768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } }
I20260928 14:58:29.852699 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588960d9fc0a4ea6bbe99b83594c8b0f. Latest consensus state: current_term: 2 leader_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } } }
I20260928 14:58:29.852775 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:29.852703 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588960d9fc0a4ea6bbe99b83594c8b0f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42277 } } }
I20260928 14:58:29.852931 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:29.853060 10772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:29.853571 10772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:29.853720 10772 catalog_manager.cc:1259] Loaded cluster ID: 482ece474f324c078703637e0303f50c
I20260928 14:58:29.853945 10772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:29.855118 10772 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:29.855297 10772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 588960d9fc0a4ea6bbe99b83594c8b0f: Loaded TSK: 0
I20260928 14:58:29.855511 10772 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:58:29.923023 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:29.923247 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:29.923286 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:29.924600 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:29.924665 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:29.924710 10766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:29.926156 10766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.193.keytab
--principal=oryx/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:32835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=32969
--tserver_master_addrs=127.1.252.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:29.926410 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:29.926630 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:29.932524 10790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.932539 10791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:29.932539 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:29.932775 10766 server_base.cc:1034] running on GCE node
I20260928 14:58:29.933032 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:29.933290 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:29.934470 10766 hybrid_clock.cc:648] HybridClock initialized: now 1790607509934435 us; error 47 us; skew 500 ppm
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:29.936959 10766 init.cc:377] Logged in from keytab as oryx/127.1.252.193@KRBTEST.COM (short username oryx)
I20260928 14:58:29.937412 10766 webserver.cc:460] Webserver started at http://127.1.252.193:32969/ using document root <none> and password file <none>
I20260928 14:58:29.937633 10766 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:29.937717 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:29.939091 10766 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:29.939769 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.939960 10766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:29.940058 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "367e2affa15b43e894bea0adee3f96e9"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "1f53c55b7ba52b1f398714f838e73fab"
server_key_iv: "bc3eebeaf9381f840f7ac12885eea2c3"
server_key_version: "encryptionkey@0"
I20260928 14:58:29.940402 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:29.968187 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:29.970557 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:29.970726 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:29.970978 10766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:29.971347 10766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:29.971411 10766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.971448 10766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:29.971475 10766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:29.977382 10766 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:32835
I20260928 14:58:29.977469 10913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:32835 every 8 connection(s)
I20260928 14:58:29.977820 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:29.984901 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10766
I20260928 14:58:29.985179 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:43651
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:42277
--webserver_port=44703
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:29.986025 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.978536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:55941)
0928 14:58:29.978720 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:29.978768 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:29.979265 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:29.979705 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:29.979709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:29.980298 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:29.980495 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.980499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.981617 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.981618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.982409 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.982412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.983026 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.983027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.983160 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.983681 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:29.983689 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:29.985884 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
W20260928 14:58:29.986266 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:29.978588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:55941 (local address 127.1.252.254:42277)
0928 14:58:29.978777 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.978786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.978862 (+ 76us) server_negotiation.cc:407] Connection header received
0928 14:58:29.979473 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.979478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.979530 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.979670 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.980625 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.981499 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.982506 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.982930 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.983052 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.986071 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:55941: BlockingRecv error: recv got EOF from 127.1.252.193:55941 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260928 14:58:29.986639 10914 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:42277 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.252.254:42277: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:29.991420 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.986921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:37741)
0928 14:58:29.986978 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:29.986988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:29.987027 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:29.987172 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:29.987173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:29.987304 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:29.987347 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.987348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.988300 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.988300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.988780 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.988781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.989141 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.989141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.989213 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.989623 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:29.989630 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:29.991320 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 14:58:29.991446 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:29.986990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:37741 (local address 127.1.252.254:42277)
0928 14:58:29.987036 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.987038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.987049 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:58:29.987073 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.987074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.987087 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.987158 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.987429 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.988207 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.988867 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.989067 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.989143 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.991405 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:37741: BlockingRecv error: recv got EOF from 127.1.252.193:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:29 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:29.996281 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:29.991727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:33339)
0928 14:58:29.991822 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:29.991834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:29.991913 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:29.992056 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:29.992056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:29.992212 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:29.992249 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.992251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.992848 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.992849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.993761 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:29.993763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.994077 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.994078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:29.994144 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.994494 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:29.994500 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:29.996240 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 14:58:29.996397 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:29.991775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:33339 (local address 127.1.252.254:42277)
0928 14:58:29.991826 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:29.991828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:29.991922 (+ 94us) server_negotiation.cc:407] Connection header received
0928 14:58:29.991974 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:29.991975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:29.991983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:29.992028 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:29.992324 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.992771 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.993837 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:29.994006 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:29.994055 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:29.996334 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:33339: BlockingRecv error: recv got EOF from 127.1.252.193:33339 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 14:58:29.996418 10914 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:58:30.070377 10918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:30.070585 10918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:30.070623 10918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:30.071775 10918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:30.071836 10918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:30.071909 10918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:30.073333 10918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.194.keytab
--principal=oryx/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=44703
--tserver_master_addrs=127.1.252.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:30.073503 10918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:30.073671 10918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:30.078601 10926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:30.078723 10923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:30.078611 10924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:30.078896 10918 server_base.cc:1034] running on GCE node
I20260928 14:58:30.079092 10918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:30.079334 10918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:30.080492 10918 hybrid_clock.cc:648] HybridClock initialized: now 1790607510080473 us; error 37 us; skew 500 ppm
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607510, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:30.082808 10918 init.cc:377] Logged in from keytab as oryx/127.1.252.194@KRBTEST.COM (short username oryx)
I20260928 14:58:30.083279 10918 webserver.cc:460] Webserver started at http://127.1.252.194:44703/ using document root <none> and password file <none>
I20260928 14:58:30.083513 10918 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:30.083592 10918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:30.084941 10918 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:30.085537 10933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:30.085698 10918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:30.085788 10918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "4ef6030c87b144aaae2b84caea28c6ba"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "e19d2a454389f26ce836ae0119901e26"
server_key_iv: "e08b8bc29f4a45101d6195b40f7f4433"
server_key_version: "encryptionkey@0"
I20260928 14:58:30.086125 10918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:30.095742 10918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:30.097627 10918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:30.097781 10918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:30.098009 10918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:30.098289 10918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:30.098340 10918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:30.098377 10918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:30.098405 10918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:30.104368 10918 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:43651
I20260928 14:58:30.104420 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:43651 every 8 connection(s)
I20260928 14:58:30.104761 10918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:30.110520 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 10918
I20260928 14:58:30.110808 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:38223
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:42277
--webserver_port=40623
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:30.111788 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.105367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:56057)
0928 14:58:30.105513 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.105575 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.106060 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.106264 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.106267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.106835 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.107017 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.107022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.107658 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.107659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.108538 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.108540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.108881 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.108882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.108983 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.109500 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:30.109508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:30.111657 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
W20260928 14:58:30.111852 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.105461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:56057 (local address 127.1.252.254:42277)
0928 14:58:30.105535 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.105537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.105655 (+ 118us) server_negotiation.cc:407] Connection header received
0928 14:58:30.106150 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.106151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.106164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.106214 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.107094 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.107571 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.108612 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.108790 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.108842 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.111765 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:56057: BlockingRecv error: recv got EOF from 127.1.252.194:56057 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 14:58:30.112082 11047 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:42277 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.252.254:42277: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:30.116914 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.112278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:34971)
0928 14:58:30.112326 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.112337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.112385 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.112519 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.112520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.112643 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.112682 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.112683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.113288 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.113289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.114145 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.114148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.114477 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.114478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.114552 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.114949 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:30.114954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:30.116829 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:58:30.116940 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.112335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:34971 (local address 127.1.252.254:42277)
0928 14:58:30.112381 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.112383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.112393 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:30.112447 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.112448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.112457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.112505 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.112756 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.113210 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.114223 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.114400 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.114456 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.116900 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:34971: BlockingRecv error: recv got EOF from 127.1.252.194:34971 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:30.121708 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.117168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:50947)
0928 14:58:30.117289 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.117299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.117349 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.117491 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.117492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.117640 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.117678 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.117680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.118247 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.118248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.119117 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.119120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.119425 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.119426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.119491 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.119827 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:30.119832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:30.121621 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
W20260928 14:58:30.121723 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.117235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:50947 (local address 127.1.252.254:42277)
0928 14:58:30.117287 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.117288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.117350 (+ 62us) server_negotiation.cc:407] Connection header received
0928 14:58:30.117414 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.117415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.117422 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.117468 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.117759 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.118174 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.119196 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.119357 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.119407 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.121688 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:50947: BlockingRecv error: recv got EOF from 127.1.252.194:50947 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 14:58:30.121819 11047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:58:30.192696 11050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:30.192901 11050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:30.192925 11050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:30.193990 11050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:30.194023 11050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:30.194049 11050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:30.195408 11050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/krb5kdc/oryx_127.1.252.195.keytab
--principal=oryx/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:38223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=40623
--tserver_master_addrs=127.1.252.254:42277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:30.195588 11050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:30.195808 11050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:30.200865 11059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:30.200875 11057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:30.200914 11056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:30.201285 11050 server_base.cc:1034] running on GCE node
I20260928 14:58:30.201515 11050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:30.201753 11050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:30.202921 11050 hybrid_clock.cc:648] HybridClock initialized: now 1790607510202910 us; error 36 us; skew 500 ppm
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607510, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:30.205350 11050 init.cc:377] Logged in from keytab as oryx/127.1.252.195@KRBTEST.COM (short username oryx)
I20260928 14:58:30.205730 11050 webserver.cc:460] Webserver started at http://127.1.252.195:40623/ using document root <none> and password file <none>
I20260928 14:58:30.205951 11050 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:30.206009 11050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:30.207281 11050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:30.207858 11066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:30.208141 11050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:30.208242 11050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "705cd72336364eccbff4b701ac81e071"
format_stamp: "Formatted at 2026-09-28 14:58:29 on dist-test-slave-sf70"
server_key: "9898d242c9428e43037610956b5c5549"
server_key_iv: "0c762c806e44d8fb63e9caadde1eb5a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:30.208544 11050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:30.219755 11050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:30.221904 11050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:30.222023 11050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:30.222294 11050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:30.222675 11050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:30.222710 11050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:30.222754 11050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:30.222781 11050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:30.228600 11050 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:38223
I20260928 14:58:30.228679 11179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:38223 every 8 connection(s)
I20260928 14:58:30.229012 11050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:58:30.236285 10917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.229751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:37365 (local address 127.1.252.254:42277)
0928 14:58:30.229816 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.229818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.230076 (+ 258us) server_negotiation.cc:407] Connection header received
0928 14:58:30.230532 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.230533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.230543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.230593 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.231551 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.232147 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.233130 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.233314 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.233370 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.236231 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:37365: BlockingRecv error: recv got EOF from 127.1.252.195:37365 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:58:30.236346 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.229752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:37365)
0928 14:58:30.229955 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.229990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.230453 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.230643 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.230646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.231214 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.231423 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.231428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.232227 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.232228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.233040 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.233043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.233389 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.233390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.233485 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.233948 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:30.233957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:30.236148 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260928 14:58:30.236362 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11050
W20260928 14:58:30.236569 11180 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:42277 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.252.254:42277: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:30.241451 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.236783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:57537)
0928 14:58:30.236833 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.236843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.236881 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.237159 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.237160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.237301 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.237340 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.237341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.238000 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.238001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.238500 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.238502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.238878 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.238879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.238940 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.239369 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:30.239375 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:30.241406 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:58:30.241600 11184 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.236926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:57537 (local address 127.1.252.254:42277)
0928 14:58:30.237034 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.237037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.237049 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:58:30.237066 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.237066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.237074 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.237155 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.237416 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.237917 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.238583 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.238785 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.238876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.241521 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:57537: BlockingRecv error: recv got EOF from 127.1.252.195:57537 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260928 14:58:30.244174 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.236899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.1.252.254:42277)
0928 14:58:30.236941 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.236943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.237066 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:58:30.237133 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.237134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.237144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.237198 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.237779 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.238233 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.238980 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.239250 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.239318 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.240680 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:30.240684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:30.240684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:30.240694 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:30.242638 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:30.243014 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:30.243018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:30.243018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:30.243133 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:30.243393 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:30.243394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:30.243394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:30.243610 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:30.243706 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:30.243968 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:58:30.244077 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:30 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:30.246193 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.241744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:57675)
0928 14:58:30.242023 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.242033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.242082 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.242235 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.242236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.242365 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.242388 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.242390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.243020 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.243022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.243573 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.243574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.243979 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.243980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.244032 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.244404 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:30.244409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:30.246118 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264}
W20260928 14:58:30.246253 11185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.241803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:57675 (local address 127.1.252.254:42277)
0928 14:58:30.241961 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.241963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.242068 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:58:30.242138 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.242139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.242149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.242234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.242480 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.242942 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.243650 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.243849 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.243963 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:30.246185 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:57675: BlockingRecv error: recv got EOF from 127.1.252.195:57675 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
W20260928 14:58:30.246320 11180 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:58:31 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:31 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:31.009167 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:30.996778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:33789)
0928 14:58:30.996943 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:30.996953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:30.997006 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:30.997136 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:30.997137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:30.997300 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:30.997352 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.997353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.998126 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.998127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.998678 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:30.998680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.999003 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.999003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:30.999043 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:31.007181 (+ 8138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:31.007186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:31.009055 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
W20260928 14:58:31.009213 11187 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:30.996872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:33789 (local address 127.1.252.254:42277)
0928 14:58:30.997005 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:30.997007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:30.997015 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:30.997057 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:30.997057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:30.997067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:30.997117 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:30.997444 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.998046 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.998749 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:30.998940 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:30.998988 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:31.009139 (+ 10151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:33789: BlockingRecv error: recv got EOF from 127.1.252.193:33789 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
Sep 28 14:58:31 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:31 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:31.126669 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:31.122154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:39729)
0928 14:58:31.122331 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:31.122347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:31.122414 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:31.122544 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:31.122545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:31.122701 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:31.122750 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:31.122752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.123452 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.123453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:31.124096 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:31.124099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.124430 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.124431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:31.124469 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:31.124909 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:31.124914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:31.126603 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":128,"threads_started":1}
W20260928 14:58:31.126796 11187 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:31.122206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:39729 (local address 127.1.252.254:42277)
0928 14:58:31.122253 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:31.122255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:31.122415 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:58:31.122458 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:31.122459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:31.122469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:31.122528 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:31.122800 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.123357 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.124170 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.124354 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.124420 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:31.126697 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:39729: BlockingRecv error: recv got EOF from 127.1.252.194:39729 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 14:58:31 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:31 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:31.251503 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:31.246711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:44837)
0928 14:58:31.246912 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:31.246922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:31.246985 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:31.247131 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:31.247132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:31.247291 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:31.247340 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:31.247341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.248026 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.248027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:31.248578 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:31.248580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.248920 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.248921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:31.248964 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:31.249430 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:31.249434 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:31.251435 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":133,"threads_started":1}
W20260928 14:58:31.251567 11187 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:31.246769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:44837 (local address 127.1.252.254:42277)
0928 14:58:31.246853 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:31.246856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:31.247009 (+ 153us) server_negotiation.cc:407] Connection header received
0928 14:58:31.247037 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:31.247038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:31.247052 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:31.247112 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:31.247412 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.247961 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.248649 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:31.248848 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:31.248903 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:31.251522 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:44837: BlockingRecv error: recv got EOF from 127.1.252.195:44837 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:58:32 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:32 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:32.014209 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:32.009613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:45169)
0928 14:58:32.009758 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:32.009768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:32.009820 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:32.009963 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:32.009963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:32.010121 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:32.010170 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:32.010172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.010914 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.010915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:32.011426 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:32.011427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.011813 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.011813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:32.011858 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:32.012293 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:32.012297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:32.014151 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260928 14:58:32.014328 11191 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:32.009684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:45169 (local address 127.1.252.254:42277)
0928 14:58:32.009849 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:32.009851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:32.009859 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:32.009879 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:32.009880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:32.009890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:32.009947 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:32.010253 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.010845 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.011492 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.011733 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.011785 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:32.014244 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:45169: BlockingRecv error: recv got EOF from 127.1.252.193:45169 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
Sep 28 14:58:32 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:32 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:32.131700 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:32.127105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:45599)
0928 14:58:32.127280 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:32.127291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:32.127351 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:32.127487 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:32.127488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:32.127645 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:32.127693 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:32.127694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.128326 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.128327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:32.128869 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:32.128870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.129179 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.129180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:32.129217 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:32.129708 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:32.129713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:32.131605 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
W20260928 14:58:32.131745 11191 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:32.127180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:45599 (local address 127.1.252.254:42277)
0928 14:58:32.127250 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:32.127252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:32.127336 (+ 84us) server_negotiation.cc:407] Connection header received
0928 14:58:32.127406 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:32.127407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:32.127417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:32.127464 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:32.127760 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.128261 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.128941 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.129117 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.129166 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:32.131684 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:45599: BlockingRecv error: recv got EOF from 127.1.252.194:45599 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 14:58:32 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:32 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:32.257025 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:32.252093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:39707)
0928 14:58:32.252254 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:32.252273 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:32.252345 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:32.252520 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:32.252521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:32.252736 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:32.252810 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:32.252813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.253775 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.253776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:32.254302 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:32.254304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.254621 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.254622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:32.254667 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:32.255085 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:32.255090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:32.256966 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
W20260928 14:58:32.257185 11191 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:32.252217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:39707 (local address 127.1.252.254:42277)
0928 14:58:32.252270 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:32.252272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:32.252327 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:58:32.252430 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:32.252431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:32.252444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:32.252495 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:32.252943 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.253709 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.254372 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:32.254553 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:32.254604 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:32.257098 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:39707: BlockingRecv error: recv got EOF from 127.1.252.195:39707 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 14:58:33 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:33 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:33.019284 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:33.014677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:51733)
0928 14:58:33.014832 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:33.014842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:33.014893 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:33.015049 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:33.015049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:33.015213 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:33.015266 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:33.015268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.015914 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.015915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:33.016450 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:33.016451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.016801 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.016803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:33.016847 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:33.017468 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:33.017473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:33.019223 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
W20260928 14:58:33.019419 11195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:33.014772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:51733 (local address 127.1.252.254:42277)
0928 14:58:33.014924 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:33.014926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:33.014934 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:33.014963 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:33.014964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:33.014973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:33.015031 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:33.015334 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.015833 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.016525 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.016725 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.016782 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:33.019329 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:51733: BlockingRecv error: recv got EOF from 127.1.252.193:51733 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
Sep 28 14:58:33 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:33 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:33.136798 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:33.132205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:34815)
0928 14:58:33.132364 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:33.132376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:33.132448 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:33.132643 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:33.132644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:33.132811 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:33.132885 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:33.132888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.133588 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.133588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:33.134104 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:33.134106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.134444 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.134446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:33.134495 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:33.134911 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:33.134916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:33.136736 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260928 14:58:33.136891 11195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:33.132391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:34815 (local address 127.1.252.254:42277)
0928 14:58:33.132453 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:33.132455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:33.132510 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:58:33.132540 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:33.132541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:33.132555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:33.132607 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:33.132969 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.133522 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.134174 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.134363 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.134414 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:33.136827 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:34815: BlockingRecv error: recv got EOF from 127.1.252.194:34815 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:58:33 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:33 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:33.262337 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:33.257443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:42961)
0928 14:58:33.257738 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:33.257749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:33.257840 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:33.258002 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:33.258003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:33.258185 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:33.258234 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:33.258236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.258901 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.258902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:33.259426 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:33.259428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.259843 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.259844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:33.259966 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:33.260427 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:33.260431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:33.262232 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":85,"threads_started":1}
W20260928 14:58:33.262415 11195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:33.257533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:42961 (local address 127.1.252.254:42277)
0928 14:58:33.257715 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:33.257717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:33.257819 (+ 102us) server_negotiation.cc:407] Connection header received
0928 14:58:33.257904 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:33.257905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:33.257920 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:33.257996 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:33.258316 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.258812 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.259515 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:33.259739 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:33.259839 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:33.262319 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:42961: BlockingRecv error: recv got EOF from 127.1.252.195:42961 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 28 14:58:34 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:34 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:34.024616 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:34.019772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:56833)
0928 14:58:34.019967 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:34.019980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:34.020040 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:34.020196 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:34.020197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:34.020387 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:34.020441 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:34.020443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.021104 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.021105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:34.021633 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:34.021634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.021984 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.021985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:34.022028 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.022542 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:34.022547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:34.024492 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":149,"threads_started":1}
W20260928 14:58:34.024686 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:34.019890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:56833 (local address 127.1.252.254:42277)
0928 14:58:34.020023 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:34.020025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:34.020034 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:34.020114 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:34.020114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:34.020127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:34.020193 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:34.020505 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.021036 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.021707 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.021917 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.021971 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.024600 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:56833: BlockingRecv error: recv got EOF from 127.1.252.193:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
Sep 28 14:58:34 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:34 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:34.142324 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:34.137280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:60189)
0928 14:58:34.137416 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:34.137427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:34.137476 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:34.137604 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:34.137605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:34.137849 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:34.137931 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:34.137934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.138830 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.138831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:34.139368 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:34.139370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.139748 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.139749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:34.139801 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.140310 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:34.140314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:34.142230 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260928 14:58:34.142373 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:34.137375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:60189 (local address 127.1.252.254:42277)
0928 14:58:34.137448 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:34.137452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:34.137466 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:58:34.137520 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:34.137521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:34.137532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:34.137585 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:34.138052 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.138741 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.139472 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.139677 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.139740 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.142308 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:60189: BlockingRecv error: recv got EOF from 127.1.252.194:60189 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:58:34 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:34 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:34.267740 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:34.262832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:45389)
0928 14:58:34.263004 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:34.263016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:34.263105 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:34.263296 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:34.263297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:34.263468 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:34.263523 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:34.263524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.264273 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.264274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:34.264798 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:34.264800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.265152 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.265153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:34.265218 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.265683 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:34.265687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:34.267675 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
W20260928 14:58:34.267931 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:34.262930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:45389 (local address 127.1.252.254:42277)
0928 14:58:34.263006 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:34.263009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:34.263086 (+ 77us) server_negotiation.cc:407] Connection header received
0928 14:58:34.263197 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:34.263197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:34.263212 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:34.263273 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:34.263603 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.264169 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.264881 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:34.265070 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:34.265150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:34.267793 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:45389: BlockingRecv error: recv got EOF from 127.1.252.195:45389 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:58:35 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:35 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:35.030336 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:35.025214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:33553)
0928 14:58:35.025337 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:35.025349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:35.025402 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:35.025569 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:35.025570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:35.025732 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:35.025782 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:35.025783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.026441 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.026442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:35.026960 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:35.026961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.027277 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.027278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:35.027319 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:35.027991 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:35.027997 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:35.030211 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
W20260928 14:58:35.030412 11203 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:35.025286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:33553 (local address 127.1.252.254:42277)
0928 14:58:35.025444 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:35.025445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:35.025454 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:35.025477 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:35.025478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:35.025487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:35.025552 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:35.025848 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.026370 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.027027 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.027210 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.027262 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:35.030337 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:33553: BlockingRecv error: recv got EOF from 127.1.252.193:33553 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
Sep 28 14:58:35 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:35 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:35.147842 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:35.142877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:38445)
0928 14:58:35.143042 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:35.143054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:35.143108 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:35.143266 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:35.143267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:35.143481 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:35.143532 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:35.143534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.144312 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.144313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:35.144922 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:35.144924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.145298 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.145299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:35.145346 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:35.145900 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:35.145905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:35.147784 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":56,"threads_started":1}
W20260928 14:58:35.148082 11203 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:35.143048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:38445 (local address 127.1.252.254:42277)
0928 14:58:35.143111 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:35.143113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:35.143150 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:58:35.143173 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:35.143174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:35.143185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:35.143259 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:35.143613 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.144230 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.145023 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.145212 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.145276 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:35.148005 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:38445: BlockingRecv error: recv got EOF from 127.1.252.194:38445 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:58:35 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:35 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:35.273779 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:35.268231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:35595)
0928 14:58:35.268387 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:35.268411 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:35.268501 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:35.268638 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:35.268639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:35.268883 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:35.268934 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:35.268936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.269745 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.269747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:35.270304 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:35.270305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.270666 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.270667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:35.270723 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:35.271278 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:35.271285 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:35.273672 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
W20260928 14:58:35.273813 11203 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:35.268284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:35595 (local address 127.1.252.254:42277)
0928 14:58:35.268398 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:35.268400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:35.268481 (+ 81us) server_negotiation.cc:407] Connection header received
0928 14:58:35.268566 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:35.268567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:35.268582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:35.268632 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:35.269043 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.269650 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.270388 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:35.270590 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:35.270666 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:35.273767 (+ 3101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:35595: BlockingRecv error: recv got EOF from 127.1.252.195:35595 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 14:58:36 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:36 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:36.042852 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.030829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:43129)
0928 14:58:36.030981 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:36.030992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:36.031048 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:36.031245 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:36.031247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:36.031459 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:36.031521 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.031523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.032289 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.032291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.033158 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.033161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.033599 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.033600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.033690 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.040873 (+ 7183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:36.040877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:36.042792 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
W20260928 14:58:36.043016 11207 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:36.030922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43129 (local address 127.1.252.254:42277)
0928 14:58:36.031028 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:36.031029 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:36.031038 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:36.031131 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:36.031132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:36.031145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:36.031218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:36.031606 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.032208 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.033240 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.033539 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.033600 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.042914 (+ 9314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:43129: BlockingRecv error: recv got EOF from 127.1.252.193:43129 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
Sep 28 14:58:36 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:36 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:36.152953 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.148344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:57207)
0928 14:58:36.148500 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:36.148510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:36.148577 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:36.148697 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:36.148698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:36.148867 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:36.148917 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.148919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.149612 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.149613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.150158 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.150160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.150526 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.150527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.150571 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.151005 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:36.151010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:36.152890 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260928 14:58:36.153048 11207 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:36.148471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:57207 (local address 127.1.252.254:42277)
0928 14:58:36.148517 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:36.148519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:36.148564 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:58:36.148625 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:36.148626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:36.148637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:36.148685 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:36.149014 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.149512 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.150249 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.150439 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.150499 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.152981 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:57207: BlockingRecv error: recv got EOF from 127.1.252.194:57207 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 14:58:36 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:36 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:36.279143 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:36.274301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:55937)
0928 14:58:36.274442 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:36.274465 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:36.274548 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:36.274731 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:36.274732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:36.274973 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:36.275039 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.275041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.275725 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.275726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.276317 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:36.276319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.276666 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.276667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:36.276717 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.277182 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:36.277187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:36.279085 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260928 14:58:36.279263 11207 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:36.274401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:55937 (local address 127.1.252.254:42277)
0928 14:58:36.274455 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:36.274457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:36.274557 (+ 100us) server_negotiation.cc:407] Connection header received
0928 14:58:36.274611 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:36.274612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:36.274628 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:36.274715 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:36.275131 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.275637 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.276397 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:36.276579 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:36.276636 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:36.279187 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:55937: BlockingRecv error: recv got EOF from 127.1.252.195:55937 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 14:58:37 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:37 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:37.048692 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.043299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:33077)
0928 14:58:37.043447 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.043461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.043532 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.043675 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.043676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.043857 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:37.043923 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.043925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.044873 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.044874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.045456 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.045458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.045838 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.045838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.045885 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.046382 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:37.046386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:37.048632 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
W20260928 14:58:37.048862 11211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:37.043395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:33077 (local address 127.1.252.254:42277)
0928 14:58:37.043537 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.043539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.043548 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:37.043601 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.043601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.043612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.043669 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:37.044041 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.044769 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.045557 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.045753 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.045826 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.048742 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:33077: BlockingRecv error: recv got EOF from 127.1.252.193:33077 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
Sep 28 14:58:37 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:37 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:37.158023 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.153403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:57419)
0928 14:58:37.153554 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.153565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.153633 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.153802 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.153802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.153958 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:37.154005 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.154006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.154675 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.154675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.155201 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.155203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.155514 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.155515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.155552 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.156058 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:37.156063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:37.157960 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
W20260928 14:58:37.158142 11211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:37.153460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:57419 (local address 127.1.252.254:42277)
0928 14:58:37.153567 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.153569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.153635 (+ 66us) server_negotiation.cc:407] Connection header received
0928 14:58:37.153707 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.153708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.153722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.153789 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:37.154070 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.154609 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.155272 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.155450 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.155500 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.158047 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:57419: BlockingRecv error: recv got EOF from 127.1.252.194:57419 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 14:58:37 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:37 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:37.284703 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:37.279638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:38391)
0928 14:58:37.279861 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:37.279873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:37.279983 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:37.280111 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:37.280112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:37.280295 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:37.280359 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.280361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.281064 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.281065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.281599 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:37.281600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.281979 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.281979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:37.282046 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.282618 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:37.282622 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:37.284608 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
W20260928 14:58:37.284754 11211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:37.279742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:38391 (local address 127.1.252.254:42277)
0928 14:58:37.279844 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:37.279847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:37.279962 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:58:37.280046 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:37.280047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:37.280060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:37.280107 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:37.280437 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.280965 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.281680 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:37.281890 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:37.281976 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:37.284681 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:38391: BlockingRecv error: recv got EOF from 127.1.252.195:38391 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 14:58:38 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:38 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:38.053925 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:38.049193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:34983)
0928 14:58:38.049370 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:38.049382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:38.049452 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:38.049600 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:38.049601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:38.049773 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:38.049823 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:38.049825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.050451 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.050451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:38.050984 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:38.050985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.051315 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.051316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:38.051356 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.051814 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:38.051818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:38.053837 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
W20260928 14:58:38.054121 11215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:38.049305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:34983 (local address 127.1.252.254:42277)
0928 14:58:38.049442 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:38.049444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:38.049454 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:38.049508 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:38.049508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:38.049520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:38.049584 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:38.049890 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.050385 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.051057 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.051249 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.051303 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.054041 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:34983: BlockingRecv error: recv got EOF from 127.1.252.193:34983 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
Sep 28 14:58:38 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:38 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:38.163455 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:38.158495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:39697)
0928 14:58:38.158634 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:38.158652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:38.158707 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:38.158836 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:38.158837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:38.158995 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:38.159045 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:38.159047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.159965 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.159967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:38.160567 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:38.160569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.160973 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.160974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:38.161019 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.161547 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:38.161552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:38.163390 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
W20260928 14:58:38.163582 11215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:38.158566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:39697 (local address 127.1.252.254:42277)
0928 14:58:38.158645 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:38.158647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:38.158696 (+ 49us) server_negotiation.cc:407] Connection header received
0928 14:58:38.158752 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:38.158753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:38.158763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:38.158815 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:38.159156 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.159827 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.160650 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.160896 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.160952 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.163480 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:39697: BlockingRecv error: recv got EOF from 127.1.252.194:39697 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 14:58:38 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:38 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:38.290366 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:38.285285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:52389)
0928 14:58:38.285499 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:38.285513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:38.285595 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:38.285753 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:38.285755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:38.285938 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:38.285987 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:38.286008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.286965 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.286965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:38.287491 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:38.287493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.287861 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.287861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:38.287971 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.288449 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:38.288453 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:38.290297 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":112,"threads_started":1}
W20260928 14:58:38.290534 11215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:38.285416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:52389 (local address 127.1.252.254:42277)
0928 14:58:38.285503 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:38.285506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:38.285587 (+ 81us) server_negotiation.cc:407] Connection header received
0928 14:58:38.285655 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:38.285656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:38.285672 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:38.285745 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:38.286145 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.286889 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.287595 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:38.287779 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:38.287854 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:38.290455 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:52389: BlockingRecv error: recv got EOF from 127.1.252.195:52389 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:58:39 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:39 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:39.058847 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:39.054354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:40167)
0928 14:58:39.054472 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:39.054486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:39.054559 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:39.054683 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:39.054684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:39.054852 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:39.054918 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:39.054920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.055564 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.055565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:39.056151 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:39.056153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.056490 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.056491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:39.056529 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:39.057006 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:39.057010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:39.058779 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
W20260928 14:58:39.058983 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:39.054415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:40167 (local address 127.1.252.254:42277)
0928 14:58:39.054566 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:39.054568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:39.054598 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:58:39.054619 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:39.054620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:39.054630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:39.054678 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:39.054988 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.055494 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.056226 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.056425 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.056479 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:39.058876 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:40167: BlockingRecv error: recv got EOF from 127.1.252.193:40167 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
Sep 28 14:58:39 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:39 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:39.168483 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:39.163926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:47025)
0928 14:58:39.164057 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:39.164069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:39.164131 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:39.164269 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:39.164269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:39.164424 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:39.164471 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:39.164473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.165120 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.165121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:39.165658 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:39.165659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.166041 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.166041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:39.166087 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:39.166513 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:39.166518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:39.168414 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260928 14:58:39.168609 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:39.163972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:47025 (local address 127.1.252.254:42277)
0928 14:58:39.164039 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:39.164041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:39.164132 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:58:39.164193 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:39.164194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:39.164205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:39.164248 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:39.164539 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.165030 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.165765 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.165954 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.166008 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:39.168509 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:47025: BlockingRecv error: recv got EOF from 127.1.252.194:47025 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 14:58:39 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:39 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:39.296090 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:39.290827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:45853)
0928 14:58:39.291046 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:39.291059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:39.291158 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:39.291328 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:39.291329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:39.291504 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:39.291578 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:39.291580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.292552 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.292553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:39.293115 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:39.293118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.293506 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.293507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:39.293557 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:39.294009 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:39.294014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:39.296028 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260928 14:58:39.296202 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:39.290942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:45853 (local address 127.1.252.254:42277)
0928 14:58:39.291043 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:39.291046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:39.291141 (+ 95us) server_negotiation.cc:407] Connection header received
0928 14:58:39.291217 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:39.291218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:39.291239 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:39.291311 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:39.291674 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.292416 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.293213 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:39.293408 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:39.293471 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:39.296126 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:45853: BlockingRecv error: recv got EOF from 127.1.252.195:45853 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 14:58:40 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:40 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:40.064180 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:40.059335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:44881)
0928 14:58:40.059511 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:40.059523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:40.059589 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:40.059723 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:40.059724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:40.059951 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:40.060012 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:40.060015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.060748 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.060786 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:40.061311 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:40.061313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.061659 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.061660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:40.061701 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:40.062232 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:40.062237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:40.064077 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
W20260928 14:58:40.064297 11223 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:40.059436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:44881 (local address 127.1.252.254:42277)
0928 14:58:40.059579 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:40.059585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:40.059593 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:40.059640 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:40.059640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:40.059651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:40.059710 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:40.060132 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.060657 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.061398 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.061591 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.061646 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:40.064203 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:44881: BlockingRecv error: recv got EOF from 127.1.252.193:44881 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Sep 28 14:58:40 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:40 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:40.174330 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:40.168989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:33551)
0928 14:58:40.169303 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:40.169316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:40.169424 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:40.169600 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:40.169601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:40.169804 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:40.169853 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:40.169855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.170569 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.170570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:40.171113 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:40.171116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.171528 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.171529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:40.171582 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:40.172145 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:40.172153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:40.174266 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":103,"threads_started":1}
W20260928 14:58:40.174499 11223 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:40.169095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:33551 (local address 127.1.252.254:42277)
0928 14:58:40.169290 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:40.169292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:40.169405 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:58:40.169490 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:40.169491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:40.169507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:40.169595 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:40.169946 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.170464 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.171235 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.171428 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.171528 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:40.174421 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:33551: BlockingRecv error: recv got EOF from 127.1.252.194:33551 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 28 14:58:40 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:40 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:40.301403 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:40.296581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:59035)
0928 14:58:40.296730 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:40.296740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:40.296790 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:40.296947 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:40.296948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:40.297115 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:40.297166 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:40.297168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.297912 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.297913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:40.298447 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:40.298449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.298815 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.298816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:40.298854 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:40.299384 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:40.299389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:40.301340 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260928 14:58:40.301676 11223 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:40.296734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:59035 (local address 127.1.252.254:42277)
0928 14:58:40.296808 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:40.296810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:40.296820 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:40.296852 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:40.296853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:40.296865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:40.296922 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:40.297272 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.297828 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.298546 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:40.298740 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:40.298807 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:40.301591 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:59035: BlockingRecv error: recv got EOF from 127.1.252.195:59035 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 14:58:41 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:41 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:41.074106 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:41.064710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:47761)
0928 14:58:41.064889 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:41.064908 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:41.064965 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:41.065090 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:41.065091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:41.065258 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:41.065306 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:41.065307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.066021 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.066021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:41.066588 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:41.066590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.067029 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.067031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:41.067096 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:41.072167 (+ 5071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:41.072172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:41.074002 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
W20260928 14:58:41.074218 11227 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:41.064761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:47761 (local address 127.1.252.254:42277)
0928 14:58:41.064932 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:41.064934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:41.064947 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:58:41.065011 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:41.065012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:41.065022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:41.065080 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:41.065370 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.065919 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.066675 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.066904 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.066995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:41.074128 (+ 7133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:47761: BlockingRecv error: recv got EOF from 127.1.252.193:47761 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
Sep 28 14:58:41 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:41 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:41.179391 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:41.174940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:34415)
0928 14:58:41.175079 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:41.175092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:41.175205 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:41.175370 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:41.175371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:41.175537 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:41.175585 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:41.175586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.176246 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.176247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:41.176788 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:41.176790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.177115 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.177116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:41.177160 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:41.177590 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:41.177595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:41.179326 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
W20260928 14:58:41.179502 11227 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:41.175034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:34415 (local address 127.1.252.254:42277)
0928 14:58:41.175083 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:41.175085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:41.175188 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:58:41.175283 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:41.175284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:41.175296 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:41.175364 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:41.175645 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.176160 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.176876 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.177051 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.177099 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:41.179427 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:34415: BlockingRecv error: recv got EOF from 127.1.252.194:34415 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 14:58:41 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:41 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:41.307190 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:41.301899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:45207)
0928 14:58:41.302088 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:41.302101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:41.302207 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:41.302357 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:41.302357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:41.302570 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:41.302617 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:41.302619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.303377 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.303378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:41.304019 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:41.304021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.304432 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.304433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:41.304504 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:41.305047 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:41.305053 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:41.307065 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
W20260928 14:58:41.307214 11227 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:41.301989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:45207 (local address 127.1.252.254:42277)
0928 14:58:41.302086 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:41.302089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:41.302185 (+ 96us) server_negotiation.cc:407] Connection header received
0928 14:58:41.302276 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:41.302277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:41.302293 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:41.302351 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:41.302708 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.303281 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.304114 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:41.304335 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:41.304429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:41.307144 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:45207: BlockingRecv error: recv got EOF from 127.1.252.195:45207 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 14:58:42 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:42 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:42.079253 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:42.074598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:40091)
0928 14:58:42.074752 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:42.074765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:42.074835 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:42.075069 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:42.075071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:42.075240 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:42.075300 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:42.075302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.075992 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.075993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:42.076530 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:42.076532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.076917 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.076918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:42.076966 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:42.077439 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:42.077443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:42.079193 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
W20260928 14:58:42.079380 11231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:42.074781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:40091 (local address 127.1.252.254:42277)
0928 14:58:42.074907 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:42.074911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:42.074923 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:58:42.074945 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:42.074946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:42.074956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:42.075019 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:42.075382 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.075919 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.076628 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.076826 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.076902 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:42.079293 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:40091: BlockingRecv error: recv got EOF from 127.1.252.193:40091 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 28 14:58:42 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:42 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:42.185051 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:42.180005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:47271)
0928 14:58:42.180161 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:42.180173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:42.180248 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:42.180434 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:42.180435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:42.180614 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:42.180667 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:42.180668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.181401 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.181402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:42.182102 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:42.182105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.182451 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.182452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:42.182494 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:42.183005 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:42.183009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:42.184975 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260928 14:58:42.185144 11231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:42.180040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:47271 (local address 127.1.252.254:42277)
0928 14:58:42.180127 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:42.180129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:42.180230 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:58:42.180308 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:42.180309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:42.180324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:42.180388 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:42.180744 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.181319 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.182179 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.182364 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.182429 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:42.185074 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:47271: BlockingRecv error: recv got EOF from 127.1.252.194:47271 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:58:42 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:42 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:42.312657 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:42.307674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:58243)
0928 14:58:42.307877 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:42.307909 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:42.308004 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:42.308135 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:42.308136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:42.308318 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:42.308365 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:42.308367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.309058 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.309059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:42.309585 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:42.309587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.309955 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.309956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:42.310012 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:42.310520 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:42.310524 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:42.312564 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":124,"threads_started":1}
W20260928 14:58:42.312695 11231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:42.307766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:58243 (local address 127.1.252.254:42277)
0928 14:58:42.307874 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:42.307876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:42.307984 (+ 108us) server_negotiation.cc:407] Connection header received
0928 14:58:42.308066 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:42.308067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:42.308081 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:42.308132 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:42.308444 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.308975 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.309669 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:42.309881 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:42.309957 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:42.312634 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:58243: BlockingRecv error: recv got EOF from 127.1.252.195:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 14:58:43 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:43 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:43.084693 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:43.080029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:42119)
0928 14:58:43.080200 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:43.080213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:43.080275 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:43.080420 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:43.080421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:43.080624 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:43.080697 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:43.080699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.081383 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.081384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:43.081920 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:43.081922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.082297 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.082298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:43.082365 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:43.082818 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:43.082822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:43.084589 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
W20260928 14:58:43.084740 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:43.080056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:42119 (local address 127.1.252.254:42277)
0928 14:58:43.080261 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:43.080263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:43.080273 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:43.080341 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:43.080341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:43.080354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:43.080416 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:43.080781 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.081295 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.082002 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.082214 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.082287 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:43.084671 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:42119: BlockingRecv error: recv got EOF from 127.1.252.193:42119 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Sep 28 14:58:43 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:43 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:43.190459 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:43.185604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:39141)
0928 14:58:43.185754 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:43.185766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:43.185825 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:43.185964 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:43.185965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:43.186126 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:43.186173 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:43.186181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.186930 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.186931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:43.187483 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:43.187484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.187824 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.187825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:43.187864 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:43.188440 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:43.188445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:43.190363 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
W20260928 14:58:43.190510 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:43.185725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:39141 (local address 127.1.252.254:42277)
0928 14:58:43.185817 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:43.185819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:43.185827 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:43.185888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:43.185889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:43.185899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:43.185941 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:43.186321 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.186852 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.187565 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.187753 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.187806 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:43.190446 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:39141: BlockingRecv error: recv got EOF from 127.1.252.194:39141 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 14:58:43 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:43 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:43.318090 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:43.313159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:53165)
0928 14:58:43.313351 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:43.313362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:43.313459 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:43.313620 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:43.313621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:43.313820 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:43.313868 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:43.313870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.314563 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.314563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:43.315138 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:43.315141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.315506 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.315507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:43.315570 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:43.316106 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:43.316110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:43.317982 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
W20260928 14:58:43.318234 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:43.313246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:53165 (local address 127.1.252.254:42277)
0928 14:58:43.313338 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:43.313340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:43.313437 (+ 97us) server_negotiation.cc:407] Connection header received
0928 14:58:43.313524 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:43.313525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:43.313547 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:43.313614 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:43.313966 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.314473 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.315237 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:43.315424 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:43.315496 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:43.318140 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:53165: BlockingRecv error: recv got EOF from 127.1.252.195:53165 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:58:44 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:44 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:44.090165 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:44.085306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:44327)
0928 14:58:44.085452 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:44.085462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:44.085517 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:44.085706 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:44.085707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:44.085969 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:44.086042 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:44.086044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.086787 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.086788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:44.087325 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:44.087327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.087663 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.087664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:44.087703 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:44.088160 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:44.088165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:44.090056 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260928 14:58:44.090212 11239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:44.085401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:44327 (local address 127.1.252.254:42277)
0928 14:58:44.085512 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:44.085514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:44.085523 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:58:44.085596 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:44.085596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:44.085609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:44.085689 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:44.086124 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.086690 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.087405 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.087600 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.087651 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:44.090137 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:44327: BlockingRecv error: recv got EOF from 127.1.252.193:44327 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
Sep 28 14:58:44 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:44 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:44.195724 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:44.190916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:41353)
0928 14:58:44.191098 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:44.191112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:44.191192 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:44.191352 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:44.191353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:44.191558 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:44.191603 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:44.191605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.192398 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.192399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:44.192946 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:44.192947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.193421 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.193422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:44.193479 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:44.193919 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:44.193924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:44.195665 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
W20260928 14:58:44.195853 11239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:44.191012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:41353 (local address 127.1.252.254:42277)
0928 14:58:44.191103 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:44.191105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:44.191169 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:58:44.191256 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:44.191257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:44.191271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:44.191346 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:44.191689 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.192302 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.193034 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.193347 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.193422 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:44.195779 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:41353: BlockingRecv error: recv got EOF from 127.1.252.194:41353 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 14:58:44 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:44 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.195@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:44.323360 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:44.318591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.195:44677)
0928 14:58:44.318741 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:44.318751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:44.318821 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:44.318958 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:44.318958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:44.319121 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:44.319170 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:44.319171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.319843 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.319844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:44.320383 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:44.320385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.320729 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.320729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:44.320767 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:44.321199 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:44.321204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:44.323304 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
W20260928 14:58:44.323508 11239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:44.318653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:44677 (local address 127.1.252.254:42277)
0928 14:58:44.318735 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:44.318737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:44.318815 (+ 78us) server_negotiation.cc:407] Connection header received
0928 14:58:44.318875 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:44.318876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:44.318888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:44.318941 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:44.319231 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.319711 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.320475 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:44.320658 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:44.320722 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:44.323423 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.195:44677: BlockingRecv error: recv got EOF from 127.1.252.195:44677 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 14:58:45 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:45 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.193@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:45.095494 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.090718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.193:49655)
0928 14:58:45.090884 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:45.090896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:45.090958 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:45.091090 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:45.091090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:45.091250 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:45.091301 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.091302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.091980 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.091981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.092517 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.092518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.092886 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.092886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.092929 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.093474 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:45.093479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:45.095434 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
W20260928 14:58:45.095602 11243 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:45.090809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:49655 (local address 127.1.252.254:42277)
0928 14:58:45.090907 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.090908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.090942 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:58:45.091000 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.091000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.091010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.091068 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.091362 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.091868 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.092594 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.092789 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.092870 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.095523 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.193:49655: BlockingRecv error: recv got EOF from 127.1.252.193:49655 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
Sep 28 14:58:45 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:58:45 dist-test-slave-sf70 krb5kdc[10183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.252.194@KRBTEST.COM for oryx/127.1.252.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:58:45.201292 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.196443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42277 (local address 127.1.252.194:54649)
0928 14:58:45.196631 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:45.196644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:45.196745 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:45.196914 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:45.196915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:45.197096 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:45.197139 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.197140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.197896 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.197897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.198470 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.198472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.198842 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.198843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.198901 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.199401 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:45.199406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:45.201224 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.252.254:42277: Server oryx/127.1.252.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
W20260928 14:58:45.201418 11243 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:45.196540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:54649 (local address 127.1.252.254:42277)
0928 14:58:45.196633 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.196635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.196730 (+ 95us) server_negotiation.cc:407] Connection header received
0928 14:58:45.196811 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.196812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.196826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.196900 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.197222 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.197797 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.198560 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.198757 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.198840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.201340 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.252.194:54649: BlockingRecv error: recv got EOF from 127.1.252.194:54649 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:58:45.240677 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10766
I20260928 14:58:45.245415 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10918
I20260928 14:58:45.249977 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11050
I20260928 14:58:45.254468 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 10696
2026-09-28T14:58:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17433 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:45Z Disabled control of system clock
I20260928 14:58:45.281836 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:43827
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:43827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:58:45.361562 11248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:45.361755 11248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:45.361775 11248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:45.362881 11248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:45.362915 11248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:45.362928 11248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:45.362941 11248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:45.362953 11248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:45.364217 11248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:43827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:43827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:45.364408 11248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:45.364636 11248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:58:45.369719 11248 server_base.cc:1034] running on GCE node
W20260928 14:58:45.369745 11253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.369902 11254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.369912 11256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:45.370292 11248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:45.370553 11248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:45.371695 11248 hybrid_clock.cc:648] HybridClock initialized: now 1790607525371671 us; error 40 us; skew 500 ppm
I20260928 14:58:45.372967 11248 webserver.cc:460] Webserver started at http://127.1.252.254:33185/ using document root <none> and password file <none>
I20260928 14:58:45.373203 11248 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:45.373250 11248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:45.373420 11248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:45.374498 11248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "177b6cca36a14e34a7db4d34115145f1"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "b8b61fd37e80f5186d0ad8b37d637487"
server_key_iv: "905e5bd06c5f45e437d84b715f189bff"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.374869 11248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "177b6cca36a14e34a7db4d34115145f1"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "b8b61fd37e80f5186d0ad8b37d637487"
server_key_iv: "905e5bd06c5f45e437d84b715f189bff"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.376372 11248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:45.377238 11262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.377447 11248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:45.377548 11248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "177b6cca36a14e34a7db4d34115145f1"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "b8b61fd37e80f5186d0ad8b37d637487"
server_key_iv: "905e5bd06c5f45e437d84b715f189bff"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.377637 11248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:45.407202 11248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:45.407545 11248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:45.407716 11248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:45.411677 11248 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:43827
I20260928 14:58:45.411708 11314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:43827 every 8 connection(s)
I20260928 14:58:45.412058 11248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:45.412738 11315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:45.415334 11315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1: Bootstrap starting.
I20260928 14:58:45.415982 11315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:45.416191 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11248
I20260928 14:58:45.416294 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:45.416369 11315 log.cc:824] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:45.416504 2035 external_mini_cluster.cc:1428] Setting key 929c35f954aadf324720f29957495ead
I20260928 14:58:45.417192 11315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1: No bootstrap required, opened a new log
I20260928 14:58:45.419389 11315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b6cca36a14e34a7db4d34115145f1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43827 } }
I20260928 14:58:45.419559 11315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:45.419601 11315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177b6cca36a14e34a7db4d34115145f1, State: Initialized, Role: FOLLOWER
I20260928 14:58:45.419737 11315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b6cca36a14e34a7db4d34115145f1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43827 } }
I20260928 14:58:45.420089 11315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:45.420140 11315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:45.420190 11315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:45.420822 11315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b6cca36a14e34a7db4d34115145f1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43827 } }
I20260928 14:58:45.420926 11315 leader_election.cc:302] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177b6cca36a14e34a7db4d34115145f1; no voters:
I20260928 14:58:45.421067 11315 leader_election.cc:288] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:45.421123 11320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:45.421307 11320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [term 1 LEADER]: Becoming Leader. State: Replica: 177b6cca36a14e34a7db4d34115145f1, State: Running, Role: LEADER
I20260928 14:58:45.421316 11315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:45.421412 11320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b6cca36a14e34a7db4d34115145f1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43827 } }
I20260928 14:58:45.421727 11320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177b6cca36a14e34a7db4d34115145f1. Latest consensus state: current_term: 1 leader_uuid: "177b6cca36a14e34a7db4d34115145f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b6cca36a14e34a7db4d34115145f1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43827 } } }
I20260928 14:58:45.421830 11320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:45.422027 11321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177b6cca36a14e34a7db4d34115145f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177b6cca36a14e34a7db4d34115145f1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 43827 } } }
I20260928 14:58:45.422039 11326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:45.422142 11321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:45.422646 11326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:45.423049 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.417022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.1.252.254:43827)
0928 14:58:45.417246 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.417255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.417294 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:58:45.417771 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.417779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.417859 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.418008 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.418188 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.419528 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.420525 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.421150 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.421642 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.422086 (+ 444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:45.422090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:45.422090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:45.422311 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:45.422354 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:45.422370 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:45.422449 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:45.422736 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 14:58:45.422812 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260928 14:58:45.424218 11326 catalog_manager.cc:1347] Generated new cluster ID: 04225c2a7b794c5885951373a16f0747
I20260928 14:58:45.424284 11326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:45.445902 11326 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:45.446584 11326 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:45.454182 11326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 177b6cca36a14e34a7db4d34115145f1: Generated new TSK 0
I20260928 14:58:45.454363 11326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:45.458909 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43827
--builtin_ntp_servers=127.1.252.212:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:58:45.542759 11339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:45.543020 11339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:45.543057 11339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:45.544271 11339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:45.544328 11339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:45.544370 11339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:45.545782 11339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:45.546028 11339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:45.546321 11339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:45.551247 11347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.551286 11345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.551276 11344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:45.551456 11339 server_base.cc:1034] running on GCE node
I20260928 14:58:45.551687 11339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:45.551929 11339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:45.553059 11339 hybrid_clock.cc:648] HybridClock initialized: now 1790607525553038 us; error 35 us; skew 500 ppm
I20260928 14:58:45.554239 11339 webserver.cc:460] Webserver started at http://127.1.252.193:42175/ using document root <none> and password file <none>
I20260928 14:58:45.554494 11339 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:45.554539 11339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:45.554680 11339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:45.555675 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "3a58d9565549496b8dd0590cc0922325"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "dcb549205bd19cfcaba91a96e48f7091"
server_key_iv: "7f236b89197e490aeca4ae6ebd8a63be"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.556033 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "3a58d9565549496b8dd0590cc0922325"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "dcb549205bd19cfcaba91a96e48f7091"
server_key_iv: "7f236b89197e490aeca4ae6ebd8a63be"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.557442 11339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:45.558243 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.558552 11339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:58:45.558652 11339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "3a58d9565549496b8dd0590cc0922325"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "dcb549205bd19cfcaba91a96e48f7091"
server_key_iv: "7f236b89197e490aeca4ae6ebd8a63be"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.558737 11339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:45.572402 11339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:45.572662 11339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:45.572798 11339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:45.572988 11339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:45.573362 11339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:45.573412 11339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.573453 11339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:45.573478 11339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.579312 11339 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:32975
I20260928 14:58:45.579353 11466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:32975 every 8 connection(s)
I20260928 14:58:45.579653 11339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:45.583505 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11339
I20260928 14:58:45.583580 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:45.583762 2035 external_mini_cluster.cc:1428] Setting key f69f630a71fbb6d6818330bccea55abb
I20260928 14:58:45.584662 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43827
--builtin_ntp_servers=127.1.252.212:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:58:45.584789 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.580327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:49665 (local address 127.1.252.254:43827)
0928 14:58:45.580372 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.580374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.580626 (+ 252us) server_negotiation.cc:407] Connection header received
0928 14:58:45.581149 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.581150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.581164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.581215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.581872 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.582397 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.583365 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.583589 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.583653 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.584532 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:45.584534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:45.584534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:45.584539 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:45.584552 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:45.584556 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:45.584611 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:45.584690 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:58:45.584713 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:58:45.585000 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.580319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43827 (local address 127.1.252.193:49665)
0928 14:58:45.580510 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:45.580541 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:45.581042 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:45.581312 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:45.581315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:45.581322 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:45.581773 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.581777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.582496 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.582497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.583266 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.583268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.583659 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.583660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.583794 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.584388 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:45.584388 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:45.584416 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:45.584437 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:45.584614 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:45.584618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:45.584621 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:45.584679 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:58:45.584778 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 14:58:45.585462 11467 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43827
I20260928 14:58:45.585548 11467 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:45.585732 11467 heartbeater.cc:511] Master 127.1.252.254:43827 requested a full tablet report, sending...
I20260928 14:58:45.586244 11279 ts_manager.cc:194] Registered new tserver with Master: 3a58d9565549496b8dd0590cc0922325 (127.1.252.193:32975)
I20260928 14:58:45.586826 11279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.193:49665
W20260928 14:58:45.664260 11470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:45.664474 11470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:45.664505 11470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:45.665601 11470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:45.665660 11470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:45.665700 11470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:45.666991 11470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:45.667238 11470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:45.667528 11470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:45.672513 11475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.672510 11478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:45.673377 11470 server_base.cc:1034] running on GCE node
W20260928 14:58:45.673902 11476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:45.674175 11470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:45.674391 11470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:45.675532 11470 hybrid_clock.cc:648] HybridClock initialized: now 1790607525675518 us; error 30 us; skew 500 ppm
I20260928 14:58:45.676833 11470 webserver.cc:460] Webserver started at http://127.1.252.194:38113/ using document root <none> and password file <none>
I20260928 14:58:45.677091 11470 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:45.677146 11470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:45.677320 11470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:45.678380 11470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "57d3bc3254344eacbdf4b85b81681cd4"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "3c413f46ef8e023ab80e05d06671e0d2"
server_key_iv: "cf0ec72c2ba9ce722d5c5db3ffc7f9f9"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.678735 11470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "57d3bc3254344eacbdf4b85b81681cd4"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "3c413f46ef8e023ab80e05d06671e0d2"
server_key_iv: "cf0ec72c2ba9ce722d5c5db3ffc7f9f9"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.680178 11470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:45.680961 11484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.681277 11470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:45.681384 11470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "57d3bc3254344eacbdf4b85b81681cd4"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "3c413f46ef8e023ab80e05d06671e0d2"
server_key_iv: "cf0ec72c2ba9ce722d5c5db3ffc7f9f9"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.681478 11470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:45.714064 11470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:45.714396 11470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:45.714550 11470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:45.714788 11470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:45.715126 11470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:45.715188 11470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.715230 11470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:45.715258 11470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.721053 11470 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42377
I20260928 14:58:45.721095 11597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42377 every 8 connection(s)
I20260928 14:58:45.721443 11470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:45.726847 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.722195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:43333 (local address 127.1.252.254:43827)
0928 14:58:45.722241 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.722243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.722486 (+ 243us) server_negotiation.cc:407] Connection header received
0928 14:58:45.723132 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.723134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.723147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.723211 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.723914 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.724412 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.725393 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.725587 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.725658 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.726506 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:45.726508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:45.726508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:45.726513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:45.726529 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:45.726535 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:45.726599 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:45.726745 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 14:58:45.726769 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:58:45.727069 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.722162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43827 (local address 127.1.252.194:43333)
0928 14:58:45.722358 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:45.722400 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:45.723012 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:45.723292 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:45.723296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:45.723313 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:45.723794 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.723799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.724510 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.724511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.725288 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.725290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.725684 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.725685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.725810 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.726332 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:45.726333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:45.726361 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:45.726382 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:45.726601 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:45.726607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:45.726610 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:45.726734 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:58:45.726832 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 14:58:45.727536 11598 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43827
I20260928 14:58:45.727609 11598 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:45.727818 11598 heartbeater.cc:511] Master 127.1.252.254:43827 requested a full tablet report, sending...
I20260928 14:58:45.728204 11279 ts_manager.cc:194] Registered new tserver with Master: 57d3bc3254344eacbdf4b85b81681cd4 (127.1.252.194:42377)
I20260928 14:58:45.728545 11279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.194:43333
I20260928 14:58:45.729929 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11470
I20260928 14:58:45.730029 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:45.730226 2035 external_mini_cluster.cc:1428] Setting key 166b156cc5a4281092242ffa4c5bcaf8
I20260928 14:58:45.731055 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43827
--builtin_ntp_servers=127.1.252.212:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:58:45.811522 11601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:45.811713 11601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:45.811734 11601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:45.812844 11601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:45.812880 11601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:45.812904 11601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:45.814113 11601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:45.814289 11601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:45.814510 11601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:45.819579 11606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.819638 11607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.819592 11609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:45.819767 11601 server_base.cc:1034] running on GCE node
I20260928 14:58:45.820078 11601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:45.820320 11601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:45.821486 11601 hybrid_clock.cc:648] HybridClock initialized: now 1790607525821472 us; error 45 us; skew 500 ppm
I20260928 14:58:45.822759 11601 webserver.cc:460] Webserver started at http://127.1.252.195:36583/ using document root <none> and password file <none>
I20260928 14:58:45.823024 11601 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:45.823096 11601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:45.823246 11601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:45.824344 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "201685d27afb49ce881be28d42a0db5f"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "17c164c4399347f0efdf329a15a02128"
server_key_iv: "0c4237d043fad0690b8889a10c2860a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.824694 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "201685d27afb49ce881be28d42a0db5f"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "17c164c4399347f0efdf329a15a02128"
server_key_iv: "0c4237d043fad0690b8889a10c2860a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.825960 11601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:45.826687 11615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.826866 11601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:45.826961 11601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "201685d27afb49ce881be28d42a0db5f"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "17c164c4399347f0efdf329a15a02128"
server_key_iv: "0c4237d043fad0690b8889a10c2860a3"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.827049 11601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:45.838784 11601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:45.839054 11601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:45.839205 11601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:45.839422 11601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:45.839738 11601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:45.839792 11601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.839830 11601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:45.839859 11601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.845727 11601 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:38821
I20260928 14:58:45.845778 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:38821 every 8 connection(s)
I20260928 14:58:45.846071 11601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:45.851186 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.846768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:53023 (local address 127.1.252.254:43827)
0928 14:58:45.846811 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.846813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.846983 (+ 170us) server_negotiation.cc:407] Connection header received
0928 14:58:45.847498 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.847500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.847512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.847580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.848349 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.848846 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.849794 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.849989 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.850054 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.850854 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:45.850855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:45.850856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:45.850860 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:45.850874 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:45.850878 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:45.850930 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:45.851106 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:58:45.851128 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:58:45.851388 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:45.846690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:43827 (local address 127.1.252.195:53023)
0928 14:58:45.846873 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:45.846914 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:45.847375 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:45.847673 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:45.847677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:45.847685 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:45.848244 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.848249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.848935 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.848936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.849696 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:45.849698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:45.850072 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:45.850072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:45.850176 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:45.850704 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:45.850705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:45.850731 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:45.850751 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:45.851014 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:45.851020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:45.851023 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:45.851092 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:58:45.851168 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260928 14:58:45.851872 11729 heartbeater.cc:347] Connected to a master server at 127.1.252.254:43827
I20260928 14:58:45.851984 11729 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:45.852109 11729 heartbeater.cc:511] Master 127.1.252.254:43827 requested a full tablet report, sending...
I20260928 14:58:45.852451 11279 ts_manager.cc:194] Registered new tserver with Master: 201685d27afb49ce881be28d42a0db5f (127.1.252.195:38821)
I20260928 14:58:45.852818 11279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.195:53023
I20260928 14:58:45.855602 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11601
I20260928 14:58:45.855679 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:45.855834 2035 external_mini_cluster.cc:1428] Setting key 3deb4eee13b96ddac5f518b03f8a0b02
I20260928 14:58:45.856989 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:58:45.858798 11319 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:45.858352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.1.252.254:43827)
0928 14:58:45.858389 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:45.858390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:45.858499 (+ 109us) server_negotiation.cc:407] Connection header received
0928 14:58:45.858567 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:45.858568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:45.858576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:45.858622 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:45.858718 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45842: BlockingRecv error: recv got EOF from 127.0.0.1:45842 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:58:45.858884 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11339
I20260928 14:58:45.863732 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11470
I20260928 14:58:45.868155 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11601
I20260928 14:58:45.872340 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11248
2026-09-28T14:58:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (616 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:45Z Disabled control of system clock
I20260928 14:58:45.898440 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35665
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:58:45.981112 11740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:45.981307 11740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:45.981328 11740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:45.982359 11740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:45.982405 11740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:45.982419 11740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:45.982431 11740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:45.982443 11740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:45.983678 11740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:35665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:45.983855 11740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:45.984139 11740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:45.988791 11746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.988816 11745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:45.988811 11748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:45.989300 11740 server_base.cc:1034] running on GCE node
I20260928 14:58:45.989529 11740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:45.989749 11740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:45.990891 11740 hybrid_clock.cc:648] HybridClock initialized: now 1790607525990880 us; error 30 us; skew 500 ppm
I20260928 14:58:45.992121 11740 webserver.cc:460] Webserver started at http://127.1.252.254:37495/ using document root <none> and password file <none>
I20260928 14:58:45.992354 11740 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:45.992399 11740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:45.992527 11740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:45.993554 11740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "61ebab3ad5bd418584a1c332afedc945"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "afd02201e449bc253c5e05748e6fa834"
server_key_iv: "ca3a61380dac6166bc9a918132a8b995"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.993924 11740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "61ebab3ad5bd418584a1c332afedc945"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "afd02201e449bc253c5e05748e6fa834"
server_key_iv: "ca3a61380dac6166bc9a918132a8b995"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.995209 11740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:45.995997 11754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:45.996169 11740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:45.996263 11740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "61ebab3ad5bd418584a1c332afedc945"
format_stamp: "Formatted at 2026-09-28 14:58:45 on dist-test-slave-sf70"
server_key: "afd02201e449bc253c5e05748e6fa834"
server_key_iv: "ca3a61380dac6166bc9a918132a8b995"
server_key_version: "encryptionkey@0"
I20260928 14:58:45.996357 11740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:45.999559 11740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:45.999811 11740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:45.999974 11740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:46.003708 11740 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:35665
I20260928 14:58:46.003743 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:35665 every 8 connection(s)
I20260928 14:58:46.004120 11740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:46.004953 11807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:46.007622 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945: Bootstrap starting.
I20260928 14:58:46.008271 11807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:46.008580 11807 log.cc:824] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:46.009231 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945: No bootstrap required, opened a new log
I20260928 14:58:46.010767 11807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ebab3ad5bd418584a1c332afedc945" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35665 } }
I20260928 14:58:46.010928 11807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:46.010977 11807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ebab3ad5bd418584a1c332afedc945, State: Initialized, Role: FOLLOWER
I20260928 14:58:46.011128 11807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ebab3ad5bd418584a1c332afedc945" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35665 } }
I20260928 14:58:46.011188 11807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:46.011229 11807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:46.011276 11807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:46.011785 11807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ebab3ad5bd418584a1c332afedc945" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35665 } }
I20260928 14:58:46.011924 11807 leader_election.cc:302] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61ebab3ad5bd418584a1c332afedc945; no voters:
I20260928 14:58:46.012096 11807 leader_election.cc:288] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:46.012127 11810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:46.012300 11810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [term 1 LEADER]: Becoming Leader. State: Replica: 61ebab3ad5bd418584a1c332afedc945, State: Running, Role: LEADER
I20260928 14:58:46.012382 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11740
I20260928 14:58:46.012408 11807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:46.012380 11810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ebab3ad5bd418584a1c332afedc945" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35665 } }
I20260928 14:58:46.012472 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:46.012684 2035 external_mini_cluster.cc:1428] Setting key 85fa082bce63960f16742f5ea445821e
I20260928 14:58:46.012655 11811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61ebab3ad5bd418584a1c332afedc945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ebab3ad5bd418584a1c332afedc945" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35665 } } }
I20260928 14:58:46.012719 11811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:46.013259 11812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61ebab3ad5bd418584a1c332afedc945. Latest consensus state: current_term: 1 leader_uuid: "61ebab3ad5bd418584a1c332afedc945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ebab3ad5bd418584a1c332afedc945" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35665 } } }
I20260928 14:58:46.013355 11812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945 [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:46.013548 11814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:46.014034 11814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:58:46.015381 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.013551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.1.252.254:35665)
0928 14:58:46.013744 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:46.013753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:46.013821 (+ 68us) server_negotiation.cc:407] Connection header received
0928 14:58:46.013936 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:46.013943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:46.013986 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:46.014371 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:46.014930 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:46.014933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:46.014933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:46.014938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:46.014978 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:46.014992 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:46.015064 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:46.015139 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:58:46.015187 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260928 14:58:46.015715 11814 catalog_manager.cc:1347] Generated new cluster ID: 4742c102aad9499d8778cb4532464034
I20260928 14:58:46.015766 11814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:46.039636 11814 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:46.039834 11814 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:46.047609 11814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61ebab3ad5bd418584a1c332afedc945: Generated new TSK 0
I20260928 14:58:46.047801 11814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:58:46.051079 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35665
--builtin_ntp_servers=127.1.252.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:58:46.133606 11831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:46.133800 11831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:46.133821 11831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:46.134958 11831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:46.134996 11831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:46.135021 11831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:46.136479 11831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:46.136721 11831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:46.137012 11831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:46.141991 11837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:46.142052 11839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:46.141997 11836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:46.142179 11831 server_base.cc:1034] running on GCE node
I20260928 14:58:46.142479 11831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:46.142714 11831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:46.143868 11831 hybrid_clock.cc:648] HybridClock initialized: now 1790607526143856 us; error 24 us; skew 500 ppm
I20260928 14:58:46.149789 11831 webserver.cc:460] Webserver started at http://127.1.252.193:45645/ using document root <none> and password file <none>
I20260928 14:58:46.150035 11831 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:46.150104 11831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:46.150257 11831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:46.151270 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "bd81944193f445e88b08c37a84889a8e"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "56d1feb97f1b6b9e3722f9e6cbd0e271"
server_key_iv: "f3e60d5ab366a12a34a077308b1bbfc0"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.151634 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "bd81944193f445e88b08c37a84889a8e"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "56d1feb97f1b6b9e3722f9e6cbd0e271"
server_key_iv: "f3e60d5ab366a12a34a077308b1bbfc0"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.153029 11831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 14:58:46.153769 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.153997 11831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:46.154096 11831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "bd81944193f445e88b08c37a84889a8e"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "56d1feb97f1b6b9e3722f9e6cbd0e271"
server_key_iv: "f3e60d5ab366a12a34a077308b1bbfc0"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.154182 11831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:46.157595 11831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:46.157841 11831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:46.157977 11831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:46.158175 11831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:46.158468 11831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:46.158520 11831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.158558 11831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:46.158586 11831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.164294 11831 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:37763
I20260928 14:58:46.164320 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:37763 every 8 connection(s)
I20260928 14:58:46.164636 11831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:58:46.165680 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11831
I20260928 14:58:46.165788 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:46.165964 2035 external_mini_cluster.cc:1428] Setting key 7cfbd493553141b41d08d3cce1fac85b
I20260928 14:58:46.166877 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35665
--builtin_ntp_servers=127.1.252.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:58:46.167712 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.165315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:54481 (local address 127.1.252.254:35665)
0928 14:58:46.165359 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:46.165361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:46.165609 (+ 248us) server_negotiation.cc:407] Connection header received
0928 14:58:46.166217 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:46.166218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:46.166223 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:46.166279 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:46.167448 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:46.167449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:46.167449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:46.167455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:46.167472 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:46.167476 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:46.167530 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:46.167628 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 14:58:46.167643 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:58:46.167945 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.165304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35665 (local address 127.1.252.193:54481)
0928 14:58:46.165490 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:46.165519 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:46.166061 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:46.166364 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:46.166368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:46.166385 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:46.167298 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:46.167298 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:46.167325 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:46.167352 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:46.167525 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:46.167530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:46.167533 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:46.167634 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:58:46.167694 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 14:58:46.168553 11959 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35665
I20260928 14:58:46.168656 11959 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:46.168794 11959 heartbeater.cc:511] Master 127.1.252.254:35665 requested a full tablet report, sending...
I20260928 14:58:46.169277 11771 ts_manager.cc:194] Registered new tserver with Master: bd81944193f445e88b08c37a84889a8e (127.1.252.193:37763)
W20260928 14:58:46.246424 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:46.246610 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:46.246631 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:46.247764 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:46.247821 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:46.247902 11962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:46.249269 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:46.249528 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:46.249778 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:46.254606 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:46.254771 11962 server_base.cc:1034] running on GCE node
W20260928 14:58:46.254640 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:46.254844 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:46.255048 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:46.255283 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:46.256444 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790607526256399 us; error 61 us; skew 500 ppm
I20260928 14:58:46.257820 11962 webserver.cc:460] Webserver started at http://127.1.252.194:34753/ using document root <none> and password file <none>
I20260928 14:58:46.258045 11962 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:46.258111 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:46.258247 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:46.259270 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "9388e264c75c4c5da7e25f34dd315099"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "2080d19ad3c1b3a22141a68570d49ca5"
server_key_iv: "1a56678b4bcec034f8df8d41ef475bb8"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.259649 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "9388e264c75c4c5da7e25f34dd315099"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "2080d19ad3c1b3a22141a68570d49ca5"
server_key_iv: "1a56678b4bcec034f8df8d41ef475bb8"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.261014 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:46.261781 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.261972 11962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:46.262068 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "9388e264c75c4c5da7e25f34dd315099"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "2080d19ad3c1b3a22141a68570d49ca5"
server_key_iv: "1a56678b4bcec034f8df8d41ef475bb8"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.262151 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:46.265498 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:46.265750 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:46.265885 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:46.266098 11962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:46.266396 11962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:46.266448 11962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.266486 11962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:46.266513 11962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.272140 11962 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:34055
I20260928 14:58:46.272215 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:34055 every 8 connection(s)
I20260928 14:58:46.272524 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:46.275346 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.273212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:46887 (local address 127.1.252.254:35665)
0928 14:58:46.273282 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:46.273284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:46.273524 (+ 240us) server_negotiation.cc:407] Connection header received
0928 14:58:46.273977 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:46.273978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:46.273984 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:46.274036 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:46.275131 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:46.275132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:46.275133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:46.275136 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:46.275150 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:46.275155 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:46.275202 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:46.275281 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:58:46.275294 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:58:46.275554 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.273216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35665 (local address 127.1.252.194:46887)
0928 14:58:46.273406 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:46.273438 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:46.273895 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:46.274107 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:46.274110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:46.274118 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:46.274996 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:46.274996 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:46.275024 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:46.275050 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:46.275213 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:46.275217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:46.275219 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:58:46.275275 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:58:46.275321 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260928 14:58:46.276036 12090 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35665
I20260928 14:58:46.276110 12090 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:46.276268 12090 heartbeater.cc:511] Master 127.1.252.254:35665 requested a full tablet report, sending...
I20260928 14:58:46.276629 11771 ts_manager.cc:194] Registered new tserver with Master: 9388e264c75c4c5da7e25f34dd315099 (127.1.252.194:34055)
I20260928 14:58:46.281629 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 11962
I20260928 14:58:46.281713 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:46.281931 2035 external_mini_cluster.cc:1428] Setting key 0aaafbb0f9eb99880b6b8caf5afeb68f
I20260928 14:58:46.282873 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35665
--builtin_ntp_servers=127.1.252.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:58:46.362439 12093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:46.362658 12093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:46.362691 12093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:46.363831 12093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:46.363893 12093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:46.363976 12093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:46.365402 12093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:46.365653 12093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:46.365957 12093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:46.371125 12099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:46.371209 12098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:46.371323 12093 server_base.cc:1034] running on GCE node
W20260928 14:58:46.371228 12101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:46.371701 12093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:46.372037 12093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:46.373193 12093 hybrid_clock.cc:648] HybridClock initialized: now 1790607526373172 us; error 37 us; skew 500 ppm
I20260928 14:58:46.374514 12093 webserver.cc:460] Webserver started at http://127.1.252.195:40357/ using document root <none> and password file <none>
I20260928 14:58:46.374748 12093 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:46.374794 12093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:46.374924 12093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:46.376044 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "eb62054bb775435d9919bc141b19296d"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "00aba852f4b06750c40adb12c0fed07a"
server_key_iv: "91bb0f4d9d7f417f365da9c6a5ae5bd2"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.376395 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "eb62054bb775435d9919bc141b19296d"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "00aba852f4b06750c40adb12c0fed07a"
server_key_iv: "91bb0f4d9d7f417f365da9c6a5ae5bd2"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.377759 12093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:46.378466 12107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.378664 12093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:46.378732 12093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "eb62054bb775435d9919bc141b19296d"
format_stamp: "Formatted at 2026-09-28 14:58:46 on dist-test-slave-sf70"
server_key: "00aba852f4b06750c40adb12c0fed07a"
server_key_iv: "91bb0f4d9d7f417f365da9c6a5ae5bd2"
server_key_version: "encryptionkey@0"
I20260928 14:58:46.378821 12093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:46.382120 12093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:46.382370 12093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:46.382516 12093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:46.382709 12093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:46.382998 12093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:46.383050 12093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.383087 12093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:46.383116 12093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:46.388924 12093 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:33825
I20260928 14:58:46.388959 12220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:33825 every 8 connection(s)
I20260928 14:58:46.389310 12093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:46.392359 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.390060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:55793 (local address 127.1.252.254:35665)
0928 14:58:46.390109 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:46.390112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:46.390339 (+ 227us) server_negotiation.cc:407] Connection header received
0928 14:58:46.390820 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:46.390822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:46.390828 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:46.390893 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:46.392093 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:46.392094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:46.392095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:58:46.392100 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:46.392117 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:58:46.392122 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:46.392195 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:46.392274 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:58:46.392290 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:58:46.392547 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:46.390045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35665 (local address 127.1.252.195:55793)
0928 14:58:46.390219 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:46.390251 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:46.390708 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:46.390961 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:46.390965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:46.390975 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:46.391925 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:58:46.391925 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:46.391963 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:58:46.392000 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:46.392192 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:46.392197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:46.392200 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:58:46.392282 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:58:46.392342 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260928 14:58:46.393013 12221 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35665
I20260928 14:58:46.393148 12221 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:46.393333 12221 heartbeater.cc:511] Master 127.1.252.254:35665 requested a full tablet report, sending...
I20260928 14:58:46.393685 11771 ts_manager.cc:194] Registered new tserver with Master: eb62054bb775435d9919bc141b19296d (127.1.252.195:33825)
I20260928 14:58:46.397322 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 12093
I20260928 14:58:46.397408 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:46.397629 2035 external_mini_cluster.cc:1428] Setting key 2a818278de9a4d7aee20f138ead4fa50
I20260928 14:58:46.398835 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:58:46.400774 11826 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:58:46.400291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.1.252.254:35665)
0928 14:58:46.400335 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:46.400337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:46.400442 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:58:46.400501 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:46.400501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:46.400508 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:46.400547 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:46.400667 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54216: BlockingRecv error: recv got EOF from 127.0.0.1:54216 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:58:46.400903 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11831
I20260928 14:58:46.406385 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11962
I20260928 14:58:46.410487 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 12093
I20260928 14:58:46.415145 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 11740
2026-09-28T14:58:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (544 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:46 dist-test-slave-sf70 krb5kdc[12231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:46 dist-test-slave-sf70 krb5kdc[12231](info): set up 2 sockets
Sep 28 14:58:46 dist-test-slave-sf70 krb5kdc[12231](info): commencing operation
krb5kdc: starting...
W20260928 14:58:47.458781 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:47 dist-test-slave-sf70 krb5kdc[12231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:58:47.583061 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41523
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:47.661932 12244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:47.662158 12244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:47.662196 12244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:47.663348 12244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:58:47.663414 12244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:47.663440 12244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:47.663461 12244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:58:47.663481 12244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:58:47.664950 12244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:47.665217 12244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:47.665450 12244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:58:47.670594 12244 server_base.cc:1034] running on GCE node
W20260928 14:58:47.670583 12249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:47.670774 12252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:47.670598 12250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:47.671211 12244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:47.671531 12244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:47.672730 12244 hybrid_clock.cc:648] HybridClock initialized: now 1790607527672690 us; error 57 us; skew 500 ppm
Sep 28 14:58:47 dist-test-slave-sf70 krb5kdc[12231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607527, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:47.675268 12244 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:58:47.675700 12244 webserver.cc:460] Webserver started at http://127.1.252.254:46097/ using document root <none> and password file <none>
I20260928 14:58:47.675987 12244 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:47.676062 12244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:47.676223 12244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:47.677291 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "61c6e2690ffc447a932ebbcdc10900bf"
format_stamp: "Formatted at 2026-09-28 14:58:47 on dist-test-slave-sf70"
server_key: "ec06b27b7fe7075b1c39942a22d8fc31"
server_key_iv: "76a970cdbb59684ad6d98d41edc5a69a"
server_key_version: "encryptionkey@0"
I20260928 14:58:47.677650 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "61c6e2690ffc447a932ebbcdc10900bf"
format_stamp: "Formatted at 2026-09-28 14:58:47 on dist-test-slave-sf70"
server_key: "ec06b27b7fe7075b1c39942a22d8fc31"
server_key_iv: "76a970cdbb59684ad6d98d41edc5a69a"
server_key_version: "encryptionkey@0"
I20260928 14:58:47.678993 12244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:58:47.679733 12259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:47.679956 12244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:47.680130 12244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "61c6e2690ffc447a932ebbcdc10900bf"
format_stamp: "Formatted at 2026-09-28 14:58:47 on dist-test-slave-sf70"
server_key: "ec06b27b7fe7075b1c39942a22d8fc31"
server_key_iv: "76a970cdbb59684ad6d98d41edc5a69a"
server_key_version: "encryptionkey@0"
I20260928 14:58:47.680251 12244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:47.687660 12244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:47.689601 12244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:47.689762 12244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:47.693687 12244 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:41523
I20260928 14:58:47.693718 12311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:41523 every 8 connection(s)
I20260928 14:58:47.694090 12244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:58:47.695103 12312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:47.697423 12312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf: Bootstrap starting.
I20260928 14:58:47.697518 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 12244
I20260928 14:58:47.697647 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:58:47.697850 2035 external_mini_cluster.cc:1428] Setting key c62c985155cd2d713613be0008f2d61b
I20260928 14:58:47.698102 12312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:47.698446 12312 log.cc:824] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:47.699122 12312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf: No bootstrap required, opened a new log
I20260928 14:58:47.700982 12312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e2690ffc447a932ebbcdc10900bf" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41523 } }
I20260928 14:58:47.701135 12312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:47.701184 12312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c6e2690ffc447a932ebbcdc10900bf, State: Initialized, Role: FOLLOWER
I20260928 14:58:47.701321 12312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e2690ffc447a932ebbcdc10900bf" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41523 } }
I20260928 14:58:47.701431 12312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:58:47.701473 12312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:58:47.701519 12312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:47.702042 12312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e2690ffc447a932ebbcdc10900bf" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41523 } }
I20260928 14:58:47.702158 12312 leader_election.cc:302] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c6e2690ffc447a932ebbcdc10900bf; no voters:
I20260928 14:58:47.702337 12312 leader_election.cc:288] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:58:47.702355 12317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:47.702559 12317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [term 1 LEADER]: Becoming Leader. State: Replica: 61c6e2690ffc447a932ebbcdc10900bf, State: Running, Role: LEADER
I20260928 14:58:47.702615 12312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:58:47.702642 12317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e2690ffc447a932ebbcdc10900bf" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41523 } }
I20260928 14:58:47.702947 12317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c6e2690ffc447a932ebbcdc10900bf. Latest consensus state: current_term: 1 leader_uuid: "61c6e2690ffc447a932ebbcdc10900bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e2690ffc447a932ebbcdc10900bf" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41523 } } }
I20260928 14:58:47.702960 12318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c6e2690ffc447a932ebbcdc10900bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e2690ffc447a932ebbcdc10900bf" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41523 } } }
I20260928 14:58:47.703037 12317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:47.703063 12318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf [sys.catalog]: This master's current role is: LEADER
I20260928 14:58:47.703235 12323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:58:47.703788 12323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:58:47 dist-test-slave-sf70 krb5kdc[12231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:47.705720 12323 catalog_manager.cc:1347] Generated new cluster ID: 64d07b12b53c4526afe99ec1e1b6a85e
I20260928 14:58:47.705773 12323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:58:47.709175 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:47.698361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.1.252.254:41523)
0928 14:58:47.698561 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:47.698570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:47.698602 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:58:47.699030 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:47.699036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:47.699078 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:47.699221 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:47.699838 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:47.700736 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:47.701637 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:47.702047 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:47.702167 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:47.705495 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:47.705499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:47.705500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:47.705511 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:47.707519 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:47.707952 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:47.707956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:47.707956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:47.708062 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:47.708268 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:47.708269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:47.708269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:47.708583 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:47.708748 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:47.708828 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:58:47.708961 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260928 14:58:47.728209 12323 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:58:47.728765 12323 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:58:47.739789 12323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf: Generated new TSK 0
I20260928 14:58:47.740000 12323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:58:47.767521 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41523
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:47.848440 12338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:47.848636 12338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:47.848658 12338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:47.849774 12338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:47.849815 12338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:47.849841 12338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:58:47.851259 12338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:47.851493 12338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:47.851763 12338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:47.856832 12343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:47.856832 12346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:47.856987 12344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:47.857080 12338 server_base.cc:1034] running on GCE node
I20260928 14:58:47.857277 12338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:47.857522 12338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:47.858664 12338 hybrid_clock.cc:648] HybridClock initialized: now 1790607527858651 us; error 30 us; skew 500 ppm
Sep 28 14:58:47 dist-test-slave-sf70 krb5kdc[12231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607527, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:47.861207 12338 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:58:47.861663 12338 webserver.cc:460] Webserver started at http://127.1.252.193:43511/ using document root <none> and password file <none>
I20260928 14:58:47.861922 12338 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:47.861997 12338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:47.862138 12338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:47.863181 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "c8abe4cb1b284c2e840191e135f90e5a"
format_stamp: "Formatted at 2026-09-28 14:58:47 on dist-test-slave-sf70"
server_key: "9fa0aa021c986d543ad975d35a8a704e"
server_key_iv: "9bfcaa310b47e966b116164edb7a1a22"
server_key_version: "encryptionkey@0"
I20260928 14:58:47.863556 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "c8abe4cb1b284c2e840191e135f90e5a"
format_stamp: "Formatted at 2026-09-28 14:58:47 on dist-test-slave-sf70"
server_key: "9fa0aa021c986d543ad975d35a8a704e"
server_key_iv: "9bfcaa310b47e966b116164edb7a1a22"
server_key_version: "encryptionkey@0"
I20260928 14:58:47.864979 12338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:58:47.865736 12353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:47.865921 12338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:58:47.866017 12338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "c8abe4cb1b284c2e840191e135f90e5a"
format_stamp: "Formatted at 2026-09-28 14:58:47 on dist-test-slave-sf70"
server_key: "9fa0aa021c986d543ad975d35a8a704e"
server_key_iv: "9bfcaa310b47e966b116164edb7a1a22"
server_key_version: "encryptionkey@0"
I20260928 14:58:47.866099 12338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:47.885113 12338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:47.887216 12338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:47.887393 12338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:47.887634 12338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:47.888003 12338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:47.888060 12338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:47.888100 12338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:47.888128 12338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:47.894044 12338 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:35163
I20260928 14:58:47.894098 12466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:35163 every 8 connection(s)
I20260928 14:58:47.894411 12338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:58:47 dist-test-slave-sf70 krb5kdc[12231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607527, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:47.902588 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 12338
I20260928 14:58:47.902762 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:58:47.902988 2035 external_mini_cluster.cc:1428] Setting key b58a802836b2477e10f35ff970a05a64
I20260928 14:58:47.904211 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:47.895188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43275 (local address 127.1.252.254:41523)
0928 14:58:47.895237 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:47.895239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:47.895434 (+ 195us) server_negotiation.cc:407] Connection header received
0928 14:58:47.896011 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:47.896012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:47.896023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:47.896082 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:47.896995 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:47.897507 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:47.898501 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:47.898700 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:47.898761 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:47.900812 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:47.900814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:47.900814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:47.900830 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:47.902565 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:47.903113 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:47.903115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:47.903115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:47.903208 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:47.903411 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:47.903413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:47.903413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:47.903550 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:47.903721 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:47.904021 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 14:58:47.904092 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:58:47.904476 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:47.895152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41523 (local address 127.1.252.193:43275)
0928 14:58:47.895318 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:47.895360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:47.895837 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:47.896134 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:47.896138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:47.896701 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:47.896897 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:47.896901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:47.897607 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:47.897609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:47.898416 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:47.898419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:47.898776 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:47.898777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:47.898882 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:47.899330 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:47.899338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:47.900692 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:47.902685 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:47.902693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:47.902695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:47.903004 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:47.903275 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:47.903275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:47.903275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:47.903367 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:47.903678 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:47.903685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:47.903852 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 14:58:47.904005 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 14:58:47.904188 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 14:58:47.904938 12467 heartbeater.cc:347] Connected to a master server at 127.1.252.254:41523
I20260928 14:58:47.905049 12467 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:47.905264 12467 heartbeater.cc:511] Master 127.1.252.254:41523 requested a full tablet report, sending...
I20260928 14:58:47.905795 12276 ts_manager.cc:194] Registered new tserver with Master: c8abe4cb1b284c2e840191e135f90e5a (127.1.252.193:35163)
I20260928 14:58:47.906396 12276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:43275
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:58:47.930826 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41523
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:48.010612 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:48.010807 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:48.010830 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:48.011958 12472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:48.011997 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:48.012023 12472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:58:48.013356 12472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:48.013530 12472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:48.013698 12472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:48.018666 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:48.018775 12478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:48.018653 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:48.018839 12472 server_base.cc:1034] running on GCE node
I20260928 14:58:48.019089 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:48.019330 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:48.020468 12472 hybrid_clock.cc:648] HybridClock initialized: now 1790607528020458 us; error 28 us; skew 500 ppm
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607528, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:48.022991 12472 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:58:48.023458 12472 webserver.cc:460] Webserver started at http://127.1.252.194:38919/ using document root <none> and password file <none>
I20260928 14:58:48.023687 12472 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:48.023741 12472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:48.023959 12472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:48.024945 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "81fc76349af34ef29c4d1d315ca6f79a"
format_stamp: "Formatted at 2026-09-28 14:58:48 on dist-test-slave-sf70"
server_key: "af8dc02c1d901b9da5228d3a3461015c"
server_key_iv: "60c75e97106a43b7747edaf9846811c9"
server_key_version: "encryptionkey@0"
I20260928 14:58:48.025296 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "81fc76349af34ef29c4d1d315ca6f79a"
format_stamp: "Formatted at 2026-09-28 14:58:48 on dist-test-slave-sf70"
server_key: "af8dc02c1d901b9da5228d3a3461015c"
server_key_iv: "60c75e97106a43b7747edaf9846811c9"
server_key_version: "encryptionkey@0"
I20260928 14:58:48.026588 12472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:48.027403 12487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:48.027570 12472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:58:48.027668 12472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "81fc76349af34ef29c4d1d315ca6f79a"
format_stamp: "Formatted at 2026-09-28 14:58:48 on dist-test-slave-sf70"
server_key: "af8dc02c1d901b9da5228d3a3461015c"
server_key_iv: "60c75e97106a43b7747edaf9846811c9"
server_key_version: "encryptionkey@0"
I20260928 14:58:48.027762 12472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:48.043114 12472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:48.045526 12472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:48.045686 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:48.045910 12472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:48.046213 12472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:48.046274 12472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:48.046315 12472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:48.046342 12472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:48.052233 12472 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42787
I20260928 14:58:48.052265 12600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42787 every 8 connection(s)
I20260928 14:58:48.052593 12472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:58:48.055253 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 12472
I20260928 14:58:48.055383 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:58:48.055584 2035 external_mini_cluster.cc:1428] Setting key 85a7ea0637ba31b78f08a7101e4b2b76
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607528, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:48.062824 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.053356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:49377 (local address 127.1.252.254:41523)
0928 14:58:48.053428 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.053431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.053568 (+ 137us) server_negotiation.cc:407] Connection header received
0928 14:58:48.054096 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.054097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.054109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.054168 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:48.055086 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.055867 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.056943 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.057254 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.057330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.059756 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:48.059758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:48.059758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:48.059766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:48.061338 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:48.061806 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:48.061808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:48.061808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:48.061871 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:48.062138 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:48.062139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:48.062140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:48.062268 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:48.062341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.062646 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.062729 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:58:48.063032 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.053243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41523 (local address 127.1.252.194:49377)
0928 14:58:48.053440 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.053483 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.053992 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.054241 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.054245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.054804 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:48.054990 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.054994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.056001 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.056002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.056855 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.056857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.057348 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.057349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.057451 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.057980 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:48.057991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:48.059628 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:48.061466 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:48.061473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:48.061475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:48.061703 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:48.061949 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:48.061949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:48.061950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:48.062066 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:48.062353 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:48.062359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:48.062502 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.062608 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.062783 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260928 14:58:48.063529 12601 heartbeater.cc:347] Connected to a master server at 127.1.252.254:41523
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
I20260928 14:58:48.063625 12601 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:48.063827 12601 heartbeater.cc:511] Master 127.1.252.254:41523 requested a full tablet report, sending...
I20260928 14:58:48.064280 12276 ts_manager.cc:194] Registered new tserver with Master: 81fc76349af34ef29c4d1d315ca6f79a (127.1.252.194:42787)
I20260928 14:58:48.064716 12276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:49377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:58:48.082516 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41523
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:58:48.161841 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:58:48.162027 12606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:58:48.162051 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:58:48.163188 12606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:58:48.163231 12606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:58:48.163256 12606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:58:48.164732 12606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:41523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:58:48.164978 12606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:58:48.165208 12606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:58:48.170137 12614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:58:48.170126 12611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:48.170260 12606 server_base.cc:1034] running on GCE node
W20260928 14:58:48.170126 12612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:58:48.170542 12606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:58:48.170799 12606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:58:48.171976 12606 hybrid_clock.cc:648] HybridClock initialized: now 1790607528171950 us; error 26 us; skew 500 ppm
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607528, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:58:48.174569 12606 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:58:48.174965 12606 webserver.cc:460] Webserver started at http://127.1.252.195:44687/ using document root <none> and password file <none>
I20260928 14:58:48.175153 12606 fs_manager.cc:359] Metadata directory not provided
I20260928 14:58:48.175192 12606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:58:48.175279 12606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:58:48.176339 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "0bb639f80c354cea8d958f49fdd0f1a9"
format_stamp: "Formatted at 2026-09-28 14:58:48 on dist-test-slave-sf70"
server_key: "3ef5703e84d81da28331945aaa4e3be5"
server_key_iv: "e7eaecdd9e67003e6c91a917a92ee5ec"
server_key_version: "encryptionkey@0"
I20260928 14:58:48.176625 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "0bb639f80c354cea8d958f49fdd0f1a9"
format_stamp: "Formatted at 2026-09-28 14:58:48 on dist-test-slave-sf70"
server_key: "3ef5703e84d81da28331945aaa4e3be5"
server_key_iv: "e7eaecdd9e67003e6c91a917a92ee5ec"
server_key_version: "encryptionkey@0"
I20260928 14:58:48.177964 12606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:58:48.178671 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:48.178905 12606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:58:48.178994 12606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "0bb639f80c354cea8d958f49fdd0f1a9"
format_stamp: "Formatted at 2026-09-28 14:58:48 on dist-test-slave-sf70"
server_key: "3ef5703e84d81da28331945aaa4e3be5"
server_key_iv: "e7eaecdd9e67003e6c91a917a92ee5ec"
server_key_version: "encryptionkey@0"
I20260928 14:58:48.179049 12606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:58:48.185777 12606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:58:48.188014 12606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:58:48.188120 12606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:58:48.188275 12606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:58:48.188583 12606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:58:48.188617 12606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:48.188638 12606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:58:48.188727 12606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:58:48.194391 12606 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35947
I20260928 14:58:48.194473 12734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35947 every 8 connection(s)
I20260928 14:58:48.194767 12606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:58:48.197250 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 12606
I20260928 14:58:48.197362 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:58:48.197561 2035 external_mini_cluster.cc:1428] Setting key 14df5a14aef23788a91bbe70806411cf
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607528, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:58:48.204854 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.195557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:51015 (local address 127.1.252.254:41523)
0928 14:58:48.195612 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.195614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.195805 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:58:48.196289 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.196291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.196301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.196360 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:48.197445 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.198131 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.199293 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.199483 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.199537 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.201663 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:48.201664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:48.201665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:48.201673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:48.203260 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:48.203692 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:48.203693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:48.203693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:48.203794 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:48.204219 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:48.204220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:48.204221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:48.204382 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:48.204440 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.204702 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.204790 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:58:48.204955 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.195503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41523 (local address 127.1.252.195:51015)
0928 14:58:48.195659 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.195699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.196201 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.196408 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.196411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.197063 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:58:48.197359 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.197365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.198226 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.198227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.199216 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.199219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.199556 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.199557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.199659 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.200237 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:58:48.200245 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:58:48.201559 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:58:48.203358 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:48.203364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:48.203366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:48.203607 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:48.203862 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:58:48.203863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:58:48.203863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:58:48.204023 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:58:48.204439 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:58:48.204444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:58:48.204562 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.204654 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.204770 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260928 14:58:48.205477 12735 heartbeater.cc:347] Connected to a master server at 127.1.252.254:41523
I20260928 14:58:48.205592 12735 heartbeater.cc:464] Registering TS with master...
I20260928 14:58:48.205767 12735 heartbeater.cc:511] Master 127.1.252.254:41523 requested a full tablet report, sending...
I20260928 14:58:48.206176 12276 ts_manager.cc:194] Registered new tserver with Master: 0bb639f80c354cea8d958f49fdd0f1a9 (127.1.252.195:35947)
I20260928 14:58:48.206506 12276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:51015
I20260928 14:58:48.209379 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:58:48.216524 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.210638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.1.252.254:41523)
0928 14:58:48.210692 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.210694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.210796 (+ 102us) server_negotiation.cc:407] Connection header received
0928 14:58:48.210848 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.210849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.210857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.210903 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:58:48.211412 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.211966 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.212753 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.212926 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.212981 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.214107 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:58:48.214108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:58:48.214108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:58:48.214114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:58:48.215444 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:48.215756 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:48.215757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:48.215757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:48.215806 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:58:48.216040 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:58:48.216041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:58:48.216041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:58:48.216169 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:58:48.216246 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.216408 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.216465 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:58:48.218137 12276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:58:48.219753 12276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:58:48.226135 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.222051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.1.252.195:35947)
0928 14:58:48.222230 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.222234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.222373 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:58:48.222465 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.222467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.222484 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.222559 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:48.222754 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.223808 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225056 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.225694 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225769 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.225831 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.225996 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.226040 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260928 14:58:48.226243 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.221897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42787 (local address 127.0.0.1:40900)
0928 14:58:48.222043 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.222079 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.222144 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.222479 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.222482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.222501 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:48.222563 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.222564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.223917 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.223918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.225234 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.225235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.225952 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.226016 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.226020 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.226087 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.226149 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":95,"thread_start_us":52,"threads_started":1}
I20260928 14:58:48.226269 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.222143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.1.252.194:42787)
0928 14:58:48.222300 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.222305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.222315 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:48.222352 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.222354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.222369 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.222473 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:48.222655 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.223790 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225334 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.225866 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225941 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.225998 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.226098 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.226143 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 14:58:48.226576 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.221897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35947 (local address 127.0.0.1:42672)
0928 14:58:48.222311 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.222321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.222396 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.222632 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.222633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.222635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:48.222691 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.222693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.223935 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.223936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.224977 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.224979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.225786 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.225867 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.225882 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.226456 (+ 574us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.226500 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":46,"thread_start_us":52,"threads_started":1}
I20260928 14:58:48.226658 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.221915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35163 (local address 127.0.0.1:34310)
0928 14:58:48.222474 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.222483 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.222526 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.222930 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.222931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.222933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:48.222966 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.222967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.224071 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.224072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.225229 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.225232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.226279 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.226280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.226472 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.226477 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.226555 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.226596 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260928 14:58:48.226986 12535 tablet_service.cc:1467] Processing CreateTablet for tablet c96f89e99b4446228664fbc82d0215fc (DEFAULT_TABLE table=test-table [id=023782f8fa5f4d748b4f84f8b779b234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:48.227350 12535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96f89e99b4446228664fbc82d0215fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:48.227324 12669 tablet_service.cc:1467] Processing CreateTablet for tablet c96f89e99b4446228664fbc82d0215fc (DEFAULT_TABLE table=test-table [id=023782f8fa5f4d748b4f84f8b779b234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:48.227617 12669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96f89e99b4446228664fbc82d0215fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:48.228094 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.222166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.1.252.193:35163)
0928 14:58:48.222337 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.222341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.222582 (+ 241us) server_negotiation.cc:407] Connection header received
0928 14:58:48.222623 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.222625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.222639 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.222852 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:48.223057 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.223806 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.225317 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.226058 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.226118 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.226169 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.227965 (+ 1796us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.228014 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260928 14:58:48.228693 12401 tablet_service.cc:1467] Processing CreateTablet for tablet c96f89e99b4446228664fbc82d0215fc (DEFAULT_TABLE table=test-table [id=023782f8fa5f4d748b4f84f8b779b234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:58:48.229015 12401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96f89e99b4446228664fbc82d0215fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:58:48.230055 12749 tablet_bootstrap.cc:492] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Bootstrap starting.
I20260928 14:58:48.230685 12749 tablet_bootstrap.cc:654] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:48.230784 12750 tablet_bootstrap.cc:492] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Bootstrap starting.
I20260928 14:58:48.231338 12749 log.cc:824] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:48.231571 12750 tablet_bootstrap.cc:654] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:48.232115 12750 log.cc:824] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:48.232293 12752 tablet_bootstrap.cc:492] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a: Bootstrap starting.
I20260928 14:58:48.232486 12749 tablet_bootstrap.cc:492] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: No bootstrap required, opened a new log
I20260928 14:58:48.232546 12749 ts_tablet_manager.cc:1397] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:58:48.232780 12750 tablet_bootstrap.cc:492] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: No bootstrap required, opened a new log
I20260928 14:58:48.232858 12750 ts_tablet_manager.cc:1397] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:58:48.232918 12752 tablet_bootstrap.cc:654] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a: Neither blocks nor log segments found. Creating new log.
I20260928 14:58:48.233397 12752 log.cc:824] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:58:48.234030 12752 tablet_bootstrap.cc:492] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a: No bootstrap required, opened a new log
I20260928 14:58:48.234107 12752 ts_tablet_manager.cc:1397] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:58:48.234481 12749 raft_consensus.cc:348] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.234658 12749 raft_consensus.cc:374] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:48.234694 12749 raft_consensus.cc:729] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fc76349af34ef29c4d1d315ca6f79a, State: Initialized, Role: FOLLOWER
I20260928 14:58:48.234726 12750 raft_consensus.cc:348] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.234799 12749 consensus_queue.cc:260] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.234872 12750 raft_consensus.cc:374] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:48.234910 12750 raft_consensus.cc:729] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb639f80c354cea8d958f49fdd0f1a9, State: Initialized, Role: FOLLOWER
I20260928 14:58:48.235018 12750 consensus_queue.cc:260] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.235065 12749 ts_tablet_manager.cc:1428] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:58:48.235193 12601 heartbeater.cc:503] Master 127.1.252.254:41523 was elected leader, sending a full tablet report...
I20260928 14:58:48.235277 12750 ts_tablet_manager.cc:1428] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:58:48.235347 12735 heartbeater.cc:503] Master 127.1.252.254:41523 was elected leader, sending a full tablet report...
I20260928 14:58:48.236454 12752 raft_consensus.cc:348] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.236613 12752 raft_consensus.cc:374] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:58:48.236646 12752 raft_consensus.cc:729] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8abe4cb1b284c2e840191e135f90e5a, State: Initialized, Role: FOLLOWER
I20260928 14:58:48.236747 12752 consensus_queue.cc:260] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.237023 12752 ts_tablet_manager.cc:1428] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:58:48.237054 12467 heartbeater.cc:503] Master 127.1.252.254:41523 was elected leader, sending a full tablet report...
W20260928 14:58:48.303233 12602 tablet.cc:2369] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:58:48.314917 12756 raft_consensus.cc:482] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:58:48.315034 12756 raft_consensus.cc:504] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.315408 12756 leader_election.cc:288] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81fc76349af34ef29c4d1d315ca6f79a (127.1.252.194:42787), c8abe4cb1b284c2e840191e135f90e5a (127.1.252.193:35163)
I20260928 14:58:48.318339 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.315580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42787 (local address 127.1.252.195:33177)
0928 14:58:48.315638 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.315650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.315692 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.315925 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.315927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.315929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:48.316001 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.316003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.316667 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.316668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.317515 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.317517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.318081 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.318082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.318168 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.318172 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.318226 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.318266 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":46}
I20260928 14:58:48.318368 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.315660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:33177 (local address 127.1.252.194:42787)
0928 14:58:48.315764 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.315766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.315776 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:58:48.315805 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.315806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.315816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.315879 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:48.316083 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.316582 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.317606 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.318006 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.318061 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.318068 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.318242 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.318281 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:58:48.318482 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.315593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35163 (local address 127.1.252.195:52005)
0928 14:58:48.315865 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.315877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:58:48.315956 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:58:48.316091 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:58:48.316092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:58:48.316093 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:58:48.316153 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.316154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.316823 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.316824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.317598 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:58:48.317600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.318193 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.318193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:58:48.318284 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.318289 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:58:48.318355 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:58:48.318395 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
I20260928 14:58:48.318497 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.315653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:52005 (local address 127.1.252.193:35163)
0928 14:58:48.315701 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.315703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.315955 (+ 252us) server_negotiation.cc:407] Connection header received
0928 14:58:48.316022 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.316023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.316033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.316080 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:58:48.316224 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.316734 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.317690 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.318094 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.318189 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.318197 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.318369 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.318413 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:58:48.318773 12555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96f89e99b4446228664fbc82d0215fc" candidate_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fc76349af34ef29c4d1d315ca6f79a" is_pre_election: true
I20260928 14:58:48.318941 12555 raft_consensus.cc:2454] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb639f80c354cea8d958f49fdd0f1a9 in term 0.
I20260928 14:58:48.318956 12421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96f89e99b4446228664fbc82d0215fc" candidate_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8abe4cb1b284c2e840191e135f90e5a" is_pre_election: true
I20260928 14:58:48.319130 12421 raft_consensus.cc:2454] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb639f80c354cea8d958f49fdd0f1a9 in term 0.
I20260928 14:58:48.319186 12623 leader_election.cc:302] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb639f80c354cea8d958f49fdd0f1a9, 81fc76349af34ef29c4d1d315ca6f79a; no voters:
I20260928 14:58:48.319351 12756 raft_consensus.cc:2781] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:58:48.319422 12756 raft_consensus.cc:482] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:58:48.319444 12756 raft_consensus.cc:3037] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:48.320222 12756 raft_consensus.cc:504] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.320366 12756 leader_election.cc:288] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [CANDIDATE]: Term 1 election: Requested vote from peers 81fc76349af34ef29c4d1d315ca6f79a (127.1.252.194:42787), c8abe4cb1b284c2e840191e135f90e5a (127.1.252.193:35163)
I20260928 14:58:48.320550 12421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96f89e99b4446228664fbc82d0215fc" candidate_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8abe4cb1b284c2e840191e135f90e5a"
I20260928 14:58:48.320565 12555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96f89e99b4446228664fbc82d0215fc" candidate_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fc76349af34ef29c4d1d315ca6f79a"
I20260928 14:58:48.320634 12555 raft_consensus.cc:3037] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:48.320634 12421 raft_consensus.cc:3037] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:58:48.321486 12555 raft_consensus.cc:2454] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb639f80c354cea8d958f49fdd0f1a9 in term 1.
I20260928 14:58:48.321487 12421 raft_consensus.cc:2454] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb639f80c354cea8d958f49fdd0f1a9 in term 1.
I20260928 14:58:48.321672 12624 leader_election.cc:302] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb639f80c354cea8d958f49fdd0f1a9, c8abe4cb1b284c2e840191e135f90e5a; no voters:
I20260928 14:58:48.321751 12756 raft_consensus.cc:2781] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:58:48.321883 12756 raft_consensus.cc:686] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb639f80c354cea8d958f49fdd0f1a9, State: Running, Role: LEADER
I20260928 14:58:48.321988 12756 consensus_queue.cc:237] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } }
I20260928 14:58:48.322817 12276 catalog_manager.cc:5495] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb639f80c354cea8d958f49fdd0f1a9 (127.1.252.195). New cstate: current_term: 1 leader_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb639f80c354cea8d958f49fdd0f1a9" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 } health_report { overall_health: UNKNOWN } } }
I20260928 14:58:48.343652 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.341215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.1.252.195:35947)
0928 14:58:48.341293 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:58:48.341295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:58:48.341303 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:58:48.341342 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:58:48.341343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:58:48.341365 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:58:48.341442 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:58:48.341603 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.342081 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.342856 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:58:48.343043 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:58:48.343118 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:58:48.343287 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:58:48.343385 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:58:48.343489 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:58:48.343513 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 14:58:48.343532 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:58:48.345456 12555 raft_consensus.cc:1261] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 1 FOLLOWER]: Refusing update from remote peer 0bb639f80c354cea8d958f49fdd0f1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:48.345456 12421 raft_consensus.cc:1261] T c96f89e99b4446228664fbc82d0215fc P c8abe4cb1b284c2e840191e135f90e5a [term 1 FOLLOWER]: Refusing update from remote peer 0bb639f80c354cea8d958f49fdd0f1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:58:48.345842 12759 consensus_queue.cc:1035] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81fc76349af34ef29c4d1d315ca6f79a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:48.346024 12756 consensus_queue.cc:1035] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8abe4cb1b284c2e840191e135f90e5a" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:58:48.348451 12763 mvcc.cc:205] Tried to move back new op lower bound from 7334328436100104192 to 7334328436007608320. Current Snapshot: MvccSnapshot[applied={T|T < 7334328436100104192}]
I20260928 14:58:48.348980 12765 mvcc.cc:205] Tried to move back new op lower bound from 7334328436100104192 to 7334328436007608320. Current Snapshot: MvccSnapshot[applied={T|T < 7334328436100104192}]
I20260928 14:58:48.350821 12275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:48.350915 12275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:58:48.351681 12275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61c6e2690ffc447a932ebbcdc10900bf: Sending DeleteTablet for 3 replicas of tablet c96f89e99b4446228664fbc82d0215fc
I20260928 14:58:48.352012 12669 tablet_service.cc:1514] Processing DeleteTablet for tablet c96f89e99b4446228664fbc82d0215fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:42672
I20260928 14:58:48.352030 12401 tablet_service.cc:1514] Processing DeleteTablet for tablet c96f89e99b4446228664fbc82d0215fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:34310
I20260928 14:58:48.352025 12535 tablet_service.cc:1514] Processing DeleteTablet for tablet c96f89e99b4446228664fbc82d0215fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:58:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:40900
I20260928 14:58:48.352229 12772 tablet_replica.cc:330] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: stopping tablet replica
I20260928 14:58:48.352314 12772 raft_consensus.cc:2229] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:58:48.352414 12772 raft_consensus.cc:2258] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:48.352531 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 12338
I20260928 14:58:48.352550 12774 tablet_replica.cc:330] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: stopping tablet replica
I20260928 14:58:48.353309 12774 raft_consensus.cc:2229] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:58:48.353431 12774 raft_consensus.cc:2258] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:58:48.353832 12774 ts_tablet_manager.cc:1905] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:48.354243 12772 ts_tablet_manager.cc:1905] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:58:48.355903 12772 ts_tablet_manager.cc:1918] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:48.355938 12774 ts_tablet_manager.cc:1918] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:58:48.355978 12772 log.cc:1196] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/c96f89e99b4446228664fbc82d0215fc
I20260928 14:58:48.356005 12774 log.cc:1196] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/c96f89e99b4446228664fbc82d0215fc
I20260928 14:58:48.356272 12774 ts_tablet_manager.cc:1939] T c96f89e99b4446228664fbc82d0215fc P 81fc76349af34ef29c4d1d315ca6f79a: Deleting consensus metadata
I20260928 14:58:48.356277 12772 ts_tablet_manager.cc:1939] T c96f89e99b4446228664fbc82d0215fc P 0bb639f80c354cea8d958f49fdd0f1a9: Deleting consensus metadata
I20260928 14:58:48.356693 12263 catalog_manager.cc:4841] TS 0bb639f80c354cea8d958f49fdd0f1a9 (127.1.252.195:35947): tablet c96f89e99b4446228664fbc82d0215fc (table test-table [id=023782f8fa5f4d748b4f84f8b779b234]) successfully deleted
I20260928 14:58:48.356701 12261 catalog_manager.cc:4841] TS 81fc76349af34ef29c4d1d315ca6f79a (127.1.252.194:42787): tablet c96f89e99b4446228664fbc82d0215fc (table test-table [id=023782f8fa5f4d748b4f84f8b779b234]) successfully deleted
W20260928 14:58:48.359661 12262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:35163 (error 108)
I20260928 14:58:48.359998 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 12472
I20260928 14:58:48.360153 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:58:48.360022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:35163 (local address 127.0.0.1:34322)
0928 14:58:48.360080 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:58:48.360116 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 14:58:48.362164 12262 catalog_manager.cc:4584] TS c8abe4cb1b284c2e840191e135f90e5a (127.1.252.193:35163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c96f89e99b4446228664fbc82d0215fc: Network error: Client connection negotiation failed: client connection to 127.1.252.193:35163: connect: Connection refused (error 111)
I20260928 14:58:48.365335 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 12606
I20260928 14:58:48.370011 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 12244
2026-09-28T14:58:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1954 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12778](info): set up 2 sockets
Sep 28 14:58:48 dist-test-slave-sf70 krb5kdc[12778](info): commencing operation
krb5kdc: starting...
W20260928 14:58:49.410472 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:58:49 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:58:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/rangeradmin_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/rangeradmin_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/rangerlookup_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/rangerlookup_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
I20260928 14:58:49.590624 2035 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWmXXpN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/postgres -l logfile start
2026-09-28 14:58:52.195 UTC [12808] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:58:52.195 UTC [12808] LOG: listening on IPv4 address "127.1.252.212", port 37465
2026-09-28 14:58:52.199 UTC [12808] LOG: listening on Unix socket "/tmp/.s.PGSQL.37465"
2026-09-28 14:58:52.220 UTC [12811] LOG: database system was shut down at 2026-09-28 14:58:50 UTC
2026-09-28 14:58:52.224 UTC [12808] LOG: database system is ready to accept connections
I20260928 14:58:53.178460 2035 mini_postgres.cc:96] Postgres bound to 37465
2026-09-28 14:58:53.184 UTC [12819] FATAL: database "slave" does not exist
127.1.252.212:37465 - accepting connections
I20260928 14:58:53.185390 2035 mini_ranger.cc:161] Starting Ranger...
I20260928 14:58:53.200868 2035 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:58:53.754961 2035 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:58:53.755050 2035 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-admin
2026-09-28 14:58:54,076 [I] DB FLAVOR :POSTGRES
2026-09-28 14:58:54,076 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:58:54,077 [I] Checking connection..
2026-09-28 14:58:54,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:58:54,352 [I] Checking connection passed.
2026-09-28 14:58:54,352 [I] --------- Verifying version history table ---------
2026-09-28 14:58:54,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:58:54,606 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:58:54,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:58:54,869 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:58:54,869 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:58:54,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:58:55.104 UTC [12883] WARNING: there is no transaction in progress
2026-09-28 14:58:55,111 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:58:55,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:58:55,371 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:58:55,371 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:58:55,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:58:55,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:58:55,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sf70.c.gcp-upstream.internal','N') ;"
2026-09-28 14:58:56,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:58:56,367 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:58:56,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:58:56,615 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:58:56,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:58:56,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:58:57,113 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:58:57,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:58:58.099 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.116 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.134 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.150 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.166 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.184 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.247 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.255 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.269 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.280 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.290 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.303 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.313 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.319 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.326 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58.332 UTC [12991] WARNING: there is no transaction in progress
2026-09-28 14:58:58,994 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:58:58,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:58:59,228 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:58:59,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:58:59,494 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:58:59,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:58:59,739 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:58:59,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:58:59,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:59:00,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sf70.c.gcp-upstream.internal';"
2026-09-28 14:59:00,515 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:59:00,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:59:00,751 [I] DB_PATCHES have already been applied
I20260928 14:59:00.757464 2035 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWmXXpN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:59:00.757560 2035 mini_ranger.cc:193] Using host: 127.1.252.212
I20260928 14:59:00.760640 2035 mini_ranger.cc:227] Ranger admin URL: http://127.1.252.212:46039
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:59:02.990125 2035 mini_ranger.cc:160] Time spent starting Ranger: real 9.805s user 0.002s sys 0.009s
Sep 28, 2026 2:59:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:59:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:59:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:59:23.434638 2035 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/rangerkms_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/rangerkms_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.252.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.252.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:59:23 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:59:23.516613 2035 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:59:23.525436 2035 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:59:24.444960 2035 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:59:24.445051 2035 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms
I20260928 14:59:24.445075 2035 mini_ranger_kms.cc:227] Using postgres at 127.1.252.212:37465
2026-09-28 14:59:24,797 [I] DB FLAVOR :POSTGRES
2026-09-28 14:59:24,798 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:59:24,798 [I] Checking connection
2026-09-28 14:59:24,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:59:25,049 [I] connection success
2026-09-28 14:59:25,049 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:59:25,049 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:59:25,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:59:25,276 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:59:25,276 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:59:25,276 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:59:25,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:37465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:59:25,552 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:59:25.994798 2035 mini_ranger_kms.cc:310] Created kms service
I20260928 14:59:26.220185 2035 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:59:26.264879 2035 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:59:26 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 14:59:26.687470 2035 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:59:26.687587 2035 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:59:26.687608 2035 mini_ranger_kms.cc:240] Using host: 127.1.252.212
I20260928 14:59:26.691295 2035 mini_ranger_kms.cc:276] Ranger KMS PID: 13172
I20260928 14:59:26.691430 2035 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.252.212:51229
Sep 28, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:59:28.891530 2035 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.375s user 0.010s sys 0.176s
I20260928 14:59:28.891642 2035 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:59:28.891753 2035 mini_ranger_kms.cc:401] 127.1.252.212:51229/kms/v1/keys
Sep 28 14:59:29 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.252.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:59:29 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.252.212@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
Sep 28 14:59:29 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.252.212@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
Sep 28 14:59:32 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 14:59:32.055539 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33717
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:33717
--ranger_config_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:51229/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:32.134888 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:32.135077 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:32.135097 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:32.136226 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:59:32.136271 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:32.136286 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:32.136298 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:59:32.136310 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:59:32.137820 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.252.212:51229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:33717
--ranger_config_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:59:32.138049 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:32.138271 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:32.143745 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:32.143857 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:32.143759 13231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:32.144004 13223 server_base.cc:1034] running on GCE node
I20260928 14:59:32.144215 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:32.144443 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:32.145584 13223 hybrid_clock.cc:648] HybridClock initialized: now 1790607572145562 us; error 37 us; skew 500 ppm
Sep 28 14:59:32 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607572, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:32.159139 13223 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 14:59:32.159593 13223 webserver.cc:460] Webserver started at http://127.1.252.254:42787/ using document root <none> and password file <none>
I20260928 14:59:32.159816 13223 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:32.159880 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:32.160079 13223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:32 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607572, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 14:59:32.208269 13223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "ca793cc62ebe48e4bee277a0b590c5ee"
format_stamp: "Formatted at 2026-09-28 14:59:32 on dist-test-slave-sf70"
server_key: "b1940253b159949a615f3492b311f4f2"
server_key_iv: "61d12e3fe0a853451c08eeca73b1c9bd"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:32.208705 13223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "ca793cc62ebe48e4bee277a0b590c5ee"
format_stamp: "Formatted at 2026-09-28 14:59:32 on dist-test-slave-sf70"
server_key: "b1940253b159949a615f3492b311f4f2"
server_key_iv: "61d12e3fe0a853451c08eeca73b1c9bd"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:32.210222 13223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:59:32.225364 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:32.225646 13223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:59:32.225751 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "ca793cc62ebe48e4bee277a0b590c5ee"
format_stamp: "Formatted at 2026-09-28 14:59:32 on dist-test-slave-sf70"
server_key: "b1940253b159949a615f3492b311f4f2"
server_key_iv: "61d12e3fe0a853451c08eeca73b1c9bd"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:32.225852 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:32.245335 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:32.247632 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:32.247792 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:32.252808 13223 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:33717
I20260928 14:59:32.252876 13291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:33717 every 8 connection(s)
I20260928 14:59:32.253226 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 14:59:32.253976 13292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:32.256438 13292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee: Bootstrap starting.
I20260928 14:59:32.257030 13292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:32.257341 13292 log.cc:824] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:32.258111 13292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee: No bootstrap required, opened a new log
I20260928 14:59:32.260519 13292 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33717 } }
I20260928 14:59:32.260672 13292 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:32.260699 13292 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca793cc62ebe48e4bee277a0b590c5ee, State: Initialized, Role: FOLLOWER
I20260928 14:59:32.260852 13292 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33717 } }
I20260928 14:59:32.260929 13292 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:59:32.260973 13292 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:59:32.261027 13292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:32.261436 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 13223
I20260928 14:59:32.261547 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 14:59:32.261647 13292 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33717 } }
I20260928 14:59:32.261725 13292 leader_election.cc:302] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca793cc62ebe48e4bee277a0b590c5ee; no voters:
I20260928 14:59:32.261739 2035 external_mini_cluster.cc:1428] Setting key 9bbe28799b73beb04b751eb8993bded8
I20260928 14:59:32.261884 13292 leader_election.cc:288] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:59:32.261917 13295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:32.262058 13295 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [term 1 LEADER]: Becoming Leader. State: Replica: ca793cc62ebe48e4bee277a0b590c5ee, State: Running, Role: LEADER
I20260928 14:59:32.262133 13295 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33717 } }
I20260928 14:59:32.262220 13292 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:59:32.262444 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33717 } } }
I20260928 14:59:32.262547 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:32.262714 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca793cc62ebe48e4bee277a0b590c5ee. Latest consensus state: current_term: 1 leader_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca793cc62ebe48e4bee277a0b590c5ee" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33717 } } }
I20260928 14:59:32.262861 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee [sys.catalog]: This master's current role is: LEADER
I20260928 14:59:32.264024 13292 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:59:32.266546 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:59:32.270354 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:59:32 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:59:32.276561 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:32.262332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.1.252.254:33717)
0928 14:59:32.262896 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:32.262906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:32.262937 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:59:32.263046 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:32.263053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:32.263107 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:32.263249 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:32.264260 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:32.267736 (+ 3476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:32.268905 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:32.269398 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:32.269523 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:32.272607 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:32.272612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:32.272612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:32.272624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:32.274717 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:32.275173 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:32.275177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:32.275177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:32.275299 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:32.275534 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:32.275534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:32.275535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:32.275857 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:32.276078 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:32.276192 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:59:32.276320 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":75,"threads_started":1}
I20260928 14:59:32.280325 13301 catalog_manager.cc:1347] Generated new cluster ID: 0299cb2b1b3d47e78892bbaa010e4f8f
I20260928 14:59:32.280381 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:59:32.296841 13301 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:59:32.297664 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:59:32.303519 13301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee: Generated new TSK 0
I20260928 14:59:32.303803 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 14:59:33.037065 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:51229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33717
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:33.167970 13338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:33.168289 13338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:33.168341 13338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:33.170250 13338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:33.170316 13338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:33.170360 13338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 14:59:33.172757 13338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.252.212:51229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:59:33.173036 13338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:33.173329 13338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:33.179118 13343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:33.179136 13344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:33.179303 13338 server_base.cc:1034] running on GCE node
W20260928 14:59:33.179105 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:33.179697 13338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:33.180027 13338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:33.181206 13338 hybrid_clock.cc:648] HybridClock initialized: now 1790607573181192 us; error 32 us; skew 500 ppm
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:33.183986 13338 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 14:59:33.184438 13338 webserver.cc:460] Webserver started at http://127.1.252.193:36395/ using document root <none> and password file <none>
I20260928 14:59:33.184682 13338 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:33.184751 13338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:33.184892 13338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 14:59:33.199627 13338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "e7a2b54943754b3b9158a78cbf972587"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "ba5ba1cfb7d546c2590177e57f48169e"
server_key_iv: "0224b913f35c5868238dce30b5ecf6b5"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.200167 13338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a2b54943754b3b9158a78cbf972587"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "ba5ba1cfb7d546c2590177e57f48169e"
server_key_iv: "0224b913f35c5868238dce30b5ecf6b5"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.202184 13338 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:59:33.220036 13353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.220264 13338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:59:33.220362 13338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "e7a2b54943754b3b9158a78cbf972587"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "ba5ba1cfb7d546c2590177e57f48169e"
server_key_iv: "0224b913f35c5868238dce30b5ecf6b5"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.220489 13338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:33.234751 13338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:33.237459 13338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:33.237632 13338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:33.237888 13338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:33.238306 13338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:33.238359 13338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.238389 13338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:33.238457 13338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.247336 13338 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:45243
I20260928 14:59:33.247588 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:45243 every 8 connection(s)
I20260928 14:59:33.247829 13338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 14:59:33.254544 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 13338
I20260928 14:59:33.254657 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 14:59:33.254844 2035 external_mini_cluster.cc:1428] Setting key 90718be59dff6ce8732b5dcf55623cb4
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:59:33.262415 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.249266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:52423 (local address 127.1.252.254:33717)
0928 14:59:33.249430 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.249432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.249649 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:59:33.250203 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.250204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.250221 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.250285 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:33.250945 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.251774 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.253319 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.253634 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.253696 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.258608 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:33.258609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:33.258610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:33.258617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:33.260475 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.260941 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.260942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.260942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.261003 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.261277 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.261278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.261279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.261412 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:33.261468 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.262214 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.262305 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260928 14:59:33.262868 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.249220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33717 (local address 127.1.252.193:52423)
0928 14:59:33.249524 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:33.249562 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:33.250073 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:33.250360 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:33.250364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:33.250551 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:33.250804 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.250810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.251924 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.251925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.253206 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.253209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.253764 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.253765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.253907 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.255167 (+ 1260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:33.255177 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:33.258502 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:33.260566 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:33.260572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:33.260575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:33.260856 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:33.261067 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:33.261067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:33.261068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:33.261195 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:33.261543 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:33.261549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:33.261719 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 14:59:33.261825 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:59:33.261977 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260928 14:59:33.263677 13470 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33717
I20260928 14:59:33.263811 13470 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:33.264046 13470 heartbeater.cc:511] Master 127.1.252.254:33717 requested a full tablet report, sending...
I20260928 14:59:33.264671 13255 ts_manager.cc:194] Registered new tserver with Master: e7a2b54943754b3b9158a78cbf972587 (127.1.252.193:45243)
I20260928 14:59:33.265578 13255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:52423
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 14:59:33.285059 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:51229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33717
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:33.397472 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:33.397801 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:33.397855 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:33.399592 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:33.399650 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:33.399688 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 14:59:33.401857 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.252.212:51229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:59:33.402108 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:33.402421 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:33.408610 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:33.408740 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:33.409199 13487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:33.409521 13476 server_base.cc:1034] running on GCE node
I20260928 14:59:33.409725 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:33.409991 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:33.411177 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790607573411148 us; error 43 us; skew 500 ppm
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:33.414283 13476 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 14:59:33.414801 13476 webserver.cc:460] Webserver started at http://127.1.252.194:44963/ using document root <none> and password file <none>
I20260928 14:59:33.415051 13476 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:33.415120 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:33.415262 13476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:33.432667 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "f522c85df4244eeb87938b7f79305a52"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "3304ff2753461c108a2b4de86d5346be"
server_key_iv: "39a443612c613f1f463c5e483eceaae5"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.434437 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "f522c85df4244eeb87938b7f79305a52"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "3304ff2753461c108a2b4de86d5346be"
server_key_iv: "39a443612c613f1f463c5e483eceaae5"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.440160 13476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 14:59:33.457950 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.458212 13476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:59:33.458302 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "f522c85df4244eeb87938b7f79305a52"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "3304ff2753461c108a2b4de86d5346be"
server_key_iv: "39a443612c613f1f463c5e483eceaae5"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.458385 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:33.484167 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:33.486498 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:33.486685 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:33.486939 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:33.487327 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:33.487402 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.487474 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:33.487530 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.494447 13476 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:36963
I20260928 14:59:33.494536 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:36963 every 8 connection(s)
I20260928 14:59:33.494835 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 14:59:33.498399 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 13476
I20260928 14:59:33.498534 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 14:59:33.498718 2035 external_mini_cluster.cc:1428] Setting key 192ed50d796c363aa00167c247796c94
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:59:33.504824 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.495565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:51263 (local address 127.1.252.254:33717)
0928 14:59:33.495626 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.495629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.495851 (+ 222us) server_negotiation.cc:407] Connection header received
0928 14:59:33.496511 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.496512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.496527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.496580 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:33.497225 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.497720 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.499138 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.499424 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.499468 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.501944 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:33.501945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:33.501946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:33.501952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:33.503368 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.503805 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.503806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.503806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.503864 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.504151 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.504152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.504152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.504277 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:33.504382 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.504648 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.504731 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:59:33.505041 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.495533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33717 (local address 127.1.252.194:51263)
0928 14:59:33.495731 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:33.495776 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:33.496414 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:33.496647 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:33.496652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:33.496878 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:33.497146 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.497151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.497826 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.497827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.499013 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.499016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.499557 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.499558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.499698 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.500408 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:33.500418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:33.501838 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:33.503469 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:33.503475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:33.503478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:33.503705 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:33.503964 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:33.503965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:33.503965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:33.504081 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:33.504373 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:33.504379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:33.504519 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 14:59:33.504629 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:59:33.504787 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":111,"threads_started":1}
I20260928 14:59:33.505615 13608 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33717
I20260928 14:59:33.505734 13608 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:33.505913 13608 heartbeater.cc:511] Master 127.1.252.254:33717 requested a full tablet report, sending...
I20260928 14:59:33.506319 13255 ts_manager.cc:194] Registered new tserver with Master: f522c85df4244eeb87938b7f79305a52 (127.1.252.194:36963)
I20260928 14:59:33.506678 13255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:51263
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 14:59:33.528911 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:51229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33717
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:59:33.640172 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:59:33.640383 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:59:33.640421 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:59:33.641584 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:59:33.641649 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:59:33.641693 13614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 14:59:33.643441 13614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.252.212:51229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 14:59:33.643687 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:59:33.643939 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:59:33.649121 13619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:33.649121 13620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:59:33.649498 13622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:59:33.652999 13614 server_base.cc:1034] running on GCE node
I20260928 14:59:33.653240 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:59:33.653507 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:59:33.654686 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790607573654638 us; error 46 us; skew 500 ppm
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:59:33.657752 13614 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 14:59:33.658313 13614 webserver.cc:460] Webserver started at http://127.1.252.195:44143/ using document root <none> and password file <none>
I20260928 14:59:33.658620 13614 fs_manager.cc:359] Metadata directory not provided
I20260928 14:59:33.658758 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:59:33.658943 13614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 14:59:33.675953 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "52caae9737c8458a9ba643f1593f1e22"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "8a976563d3e186fa749137709a5fa87c"
server_key_iv: "eb0d2ba844b599aaa87f443f3206a2d9"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.676416 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "52caae9737c8458a9ba643f1593f1e22"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "8a976563d3e186fa749137709a5fa87c"
server_key_iv: "eb0d2ba844b599aaa87f443f3206a2d9"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.678318 13614 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:59:33.690817 13630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.690986 13614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:59:33.691072 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "52caae9737c8458a9ba643f1593f1e22"
format_stamp: "Formatted at 2026-09-28 14:59:33 on dist-test-slave-sf70"
server_key: "8a976563d3e186fa749137709a5fa87c"
server_key_iv: "eb0d2ba844b599aaa87f443f3206a2d9"
server_key_version: "kuduclusterkey@0"
I20260928 14:59:33.691195 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:59:33.734794 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:59:33.737190 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:59:33.737342 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:59:33.737540 13614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:59:33.737874 13614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:59:33.737918 13614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.737951 13614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:59:33.737969 13614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:59:33.744973 13614 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:42103
I20260928 14:59:33.745440 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 14:59:33.746862 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:42103 every 8 connection(s)
I20260928 14:59:33.748878 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 13614
I20260928 14:59:33.748980 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 14:59:33.749186 2035 external_mini_cluster.cc:1428] Setting key a0bd4f49f9cbacd05ebb1d5ab0758256
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 14:59:33.756989 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.746731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:46067 (local address 127.1.252.254:33717)
0928 14:59:33.746776 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.746779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.747152 (+ 373us) server_negotiation.cc:407] Connection header received
0928 14:59:33.747671 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.747672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.747684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.747744 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:33.748442 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.749152 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.750815 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.751114 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.751156 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.753698 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:33.753700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:33.753700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:33.753709 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:33.755344 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.755941 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.755942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.755943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.756005 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.756334 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.756335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.756335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.756469 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:33.756527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.756816 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.756900 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:59:33.757383 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.746459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33717 (local address 127.1.252.195:46067)
0928 14:59:33.747012 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:33.747052 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:33.747562 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:33.747860 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:33.747863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:33.748097 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:59:33.748323 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.748329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.749279 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.749280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.750712 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.750715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.751242 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.751243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.751379 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.751973 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:59:33.751984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:59:33.753570 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:59:33.755524 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:33.755530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:33.755532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:33.755818 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:33.756081 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:59:33.756081 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:59:33.756082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:59:33.756249 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:59:33.756556 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:59:33.756561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:59:33.756674 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:59:33.756780 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:59:33.756921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":80,"threads_started":1}
I20260928 14:59:33.757845 13744 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33717
I20260928 14:59:33.757961 13744 heartbeater.cc:464] Registering TS with master...
I20260928 14:59:33.758226 13744 heartbeater.cc:511] Master 127.1.252.254:33717 requested a full tablet report, sending...
I20260928 14:59:33.758586 13255 ts_manager.cc:194] Registered new tserver with Master: 52caae9737c8458a9ba643f1593f1e22 (127.1.252.195:42103)
I20260928 14:59:33.758968 13255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:46067
I20260928 14:59:33.760954 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:59:33.769722 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.762561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.1.252.254:33717)
0928 14:59:33.762601 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.762602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.762784 (+ 182us) server_negotiation.cc:407] Connection header received
0928 14:59:33.762827 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.762828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.762838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.762892 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:59:33.763688 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.764235 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.764971 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.765203 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.765265 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.766748 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:59:33.766750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:59:33.766750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:59:33.766757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:59:33.768432 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.768871 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.768872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.768872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.768931 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:59:33.769148 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:59:33.769149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:59:33.769149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:59:33.769260 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:59:33.769365 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.769564 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.769639 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:59:33.771597 13255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:59:33.773613 13255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:59:33.781282 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.776686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.1.252.193:45243)
0928 14:59:33.776830 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.776835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.776846 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:59:33.776893 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.776895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.776912 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.776991 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:33.777231 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.778294 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.780028 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.780772 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.780966 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.781031 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.781152 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.781205 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
I20260928 14:59:33.781298 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.776142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:45243 (local address 127.0.0.1:46928)
0928 14:59:33.776267 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:33.776305 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:33.776361 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:33.777075 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:33.777078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:33.777080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:33.777144 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.777146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.778383 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.778384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.779935 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.779937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.780867 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.780868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.780947 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.781047 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 14:59:33.781139 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:59:33.781225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":63,"thread_start_us":75,"threads_started":1}
I20260928 14:59:33.782198 13398 tablet_service.cc:1467] Processing CreateTablet for tablet 6cad1595615d4f2597ca9fe9dcbadfd4 (DEFAULT_TABLE table=test-table [id=244685f3dac14e06a1f7bec91edc76ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:33.782572 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cad1595615d4f2597ca9fe9dcbadfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:33.783310 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.776357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36963 (local address 127.0.0.1:47338)
0928 14:59:33.777541 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:33.777551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:33.777591 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:33.777968 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:33.777969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:33.777970 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:33.778006 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.778007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.780888 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.780889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.782052 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.782055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.783083 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.783084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.783153 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.783156 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:59:33.783207 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:59:33.783244 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":72,"threads_started":1}
I20260928 14:59:33.783588 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.777147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.1.252.194:36963)
0928 14:59:33.777769 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.777773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.777781 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:59:33.777817 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.777819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.777832 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.777901 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:33.778069 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.779090 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.782179 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.782913 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.782980 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.783046 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.783464 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.783506 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":81,"threads_started":1}
I20260928 14:59:33.784056 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.776549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:42103 (local address 127.0.0.1:50566)
0928 14:59:33.779259 (+ 2710us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:33.779269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:33.779316 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:33.779575 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:33.779576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:33.779578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:33.779609 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.779611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.780846 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.780847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.782713 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:33.782716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.783787 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.783788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:33.783849 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.783852 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:59:33.783945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:59:33.783982 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"thread_start_us":72,"threads_started":1}
I20260928 14:59:33.784176 13531 tablet_service.cc:1467] Processing CreateTablet for tablet 6cad1595615d4f2597ca9fe9dcbadfd4 (DEFAULT_TABLE table=test-table [id=244685f3dac14e06a1f7bec91edc76ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:33.784500 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cad1595615d4f2597ca9fe9dcbadfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:33.784565 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:33.776967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.1.252.195:42103)
0928 14:59:33.777694 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:33.777698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:33.779370 (+ 1672us) server_negotiation.cc:407] Connection header received
0928 14:59:33.779408 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:33.779410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:33.779451 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:33.779511 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:33.779678 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.780731 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.782837 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:33.783598 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:33.783678 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:33.783751 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:33.784152 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 14:59:33.784203 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":83,"threads_started":1}
I20260928 14:59:33.786962 13762 tablet_bootstrap.cc:492] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: Bootstrap starting.
I20260928 14:59:33.787711 13762 tablet_bootstrap.cc:654] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:33.788125 13762 log.cc:824] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:33.788498 13763 tablet_bootstrap.cc:492] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: Bootstrap starting.
I20260928 14:59:33.788493 13676 tablet_service.cc:1467] Processing CreateTablet for tablet 6cad1595615d4f2597ca9fe9dcbadfd4 (DEFAULT_TABLE table=test-table [id=244685f3dac14e06a1f7bec91edc76ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:59:33.788800 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cad1595615d4f2597ca9fe9dcbadfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:59:33.789644 13763 tablet_bootstrap.cc:654] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:33.790131 13763 log.cc:824] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:33.790921 13762 tablet_bootstrap.cc:492] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: No bootstrap required, opened a new log
I20260928 14:59:33.791033 13762 ts_tablet_manager.cc:1397] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 14:59:33.792418 13766 tablet_bootstrap.cc:492] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Bootstrap starting.
I20260928 14:59:33.793368 13766 tablet_bootstrap.cc:654] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Neither blocks nor log segments found. Creating new log.
I20260928 14:59:33.793661 13766 log.cc:824] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Log is configured to *not* fsync() on all Append() calls
I20260928 14:59:33.794098 13762 raft_consensus.cc:348] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:33.794395 13766 tablet_bootstrap.cc:492] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: No bootstrap required, opened a new log
I20260928 14:59:33.794474 13766 ts_tablet_manager.cc:1397] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:59:33.794482 13762 raft_consensus.cc:374] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:33.794620 13762 raft_consensus.cc:729] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a2b54943754b3b9158a78cbf972587, State: Initialized, Role: FOLLOWER
I20260928 14:59:33.794811 13762 consensus_queue.cc:260] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:33.795135 13762 ts_tablet_manager.cc:1428] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 14:59:33.795284 13470 heartbeater.cc:503] Master 127.1.252.254:33717 was elected leader, sending a full tablet report...
I20260928 14:59:33.796478 13766 raft_consensus.cc:348] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:33.796627 13766 raft_consensus.cc:374] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:33.796661 13766 raft_consensus.cc:729] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52caae9737c8458a9ba643f1593f1e22, State: Initialized, Role: FOLLOWER
I20260928 14:59:33.796761 13766 consensus_queue.cc:260] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:33.797015 13766 ts_tablet_manager.cc:1428] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 14:59:33.797142 13744 heartbeater.cc:503] Master 127.1.252.254:33717 was elected leader, sending a full tablet report...
I20260928 14:59:33.800942 13763 tablet_bootstrap.cc:492] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: No bootstrap required, opened a new log
I20260928 14:59:33.801029 13763 ts_tablet_manager.cc:1397] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 14:59:33.802613 13763 raft_consensus.cc:348] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:33.802757 13763 raft_consensus.cc:374] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:59:33.802798 13763 raft_consensus.cc:729] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f522c85df4244eeb87938b7f79305a52, State: Initialized, Role: FOLLOWER
I20260928 14:59:33.802925 13763 consensus_queue.cc:260] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:33.803170 13763 ts_tablet_manager.cc:1428] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:59:33.803274 13608 heartbeater.cc:503] Master 127.1.252.254:33717 was elected leader, sending a full tablet report...
Sep 28 14:59:33 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
W20260928 14:59:33.995541 13609 tablet.cc:2369] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:59:33.996153 13745 tablet.cc:2369] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:59:33.999571 13471 tablet.cc:2369] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:59:34 dist-test-slave-sf70 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 14:59:34.150635 13770 raft_consensus.cc:482] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:59:34.150791 13770 raft_consensus.cc:504] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:34.151217 13770 leader_election.cc:288] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7a2b54943754b3b9158a78cbf972587 (127.1.252.193:45243), f522c85df4244eeb87938b7f79305a52 (127.1.252.194:36963)
I20260928 14:59:34.155175 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:34.151372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:45243 (local address 127.1.252.195:59539)
0928 14:59:34.151420 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:34.151436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:34.151502 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:34.151935 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:34.151936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:34.151939 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:34.152012 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:34.152015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.152935 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.152936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:34.154052 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:34.154055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.154910 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.154911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:34.155002 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:34.155006 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:59:34.155054 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:59:34.155091 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 14:59:34.155442 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:34.151714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:59539 (local address 127.1.252.193:45243)
0928 14:59:34.151761 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:34.151763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:34.151772 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:59:34.151814 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:34.151815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:34.151829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:34.151916 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:34.152105 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.152837 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.154151 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.154791 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.154868 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:34.154880 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:34.155338 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 14:59:34.155377 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:59:34.155772 13405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cad1595615d4f2597ca9fe9dcbadfd4" candidate_uuid: "52caae9737c8458a9ba643f1593f1e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a2b54943754b3b9158a78cbf972587" is_pre_election: true
I20260928 14:59:34.155979 13405 raft_consensus.cc:2454] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52caae9737c8458a9ba643f1593f1e22 in term 0.
I20260928 14:59:34.156231 13632 leader_election.cc:302] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52caae9737c8458a9ba643f1593f1e22, e7a2b54943754b3b9158a78cbf972587; no voters:
I20260928 14:59:34.156368 13770 raft_consensus.cc:2781] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:59:34.156442 13770 raft_consensus.cc:482] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:59:34.156471 13770 raft_consensus.cc:3037] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:34.157415 13770 raft_consensus.cc:504] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:34.157538 13770 leader_election.cc:288] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [CANDIDATE]: Term 1 election: Requested vote from peers e7a2b54943754b3b9158a78cbf972587 (127.1.252.193:45243), f522c85df4244eeb87938b7f79305a52 (127.1.252.194:36963)
I20260928 14:59:34.157719 13405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cad1595615d4f2597ca9fe9dcbadfd4" candidate_uuid: "52caae9737c8458a9ba643f1593f1e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a2b54943754b3b9158a78cbf972587"
I20260928 14:59:34.157821 13405 raft_consensus.cc:3037] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:34.158174 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:34.151659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:56659 (local address 127.1.252.194:36963)
0928 14:59:34.151706 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:34.151708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:34.154915 (+ 3207us) server_negotiation.cc:407] Connection header received
0928 14:59:34.155017 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:34.155018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:34.155031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:34.155062 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:59:34.155285 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.155961 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.157269 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.157811 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.157880 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:34.157889 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:34.158071 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:59:34.158111 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:59:34.158432 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:34.151496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36963 (local address 127.1.252.195:56659)
0928 14:59:34.154850 (+ 3354us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:34.154861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:34.154942 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:34.155122 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:59:34.155123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:59:34.155125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:59:34.155185 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:34.155186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.156060 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.156060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:34.157182 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:59:34.157186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.157894 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.157895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:59:34.157970 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:34.157976 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:59:34.158302 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 14:59:34.158345 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3333,"thread_start_us":80,"threads_started":1}
I20260928 14:59:34.158780 13405 raft_consensus.cc:2454] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52caae9737c8458a9ba643f1593f1e22 in term 1.
I20260928 14:59:34.158737 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cad1595615d4f2597ca9fe9dcbadfd4" candidate_uuid: "52caae9737c8458a9ba643f1593f1e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f522c85df4244eeb87938b7f79305a52" is_pre_election: true
I20260928 14:59:34.158886 13546 raft_consensus.cc:2454] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52caae9737c8458a9ba643f1593f1e22 in term 0.
I20260928 14:59:34.159003 13555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cad1595615d4f2597ca9fe9dcbadfd4" candidate_uuid: "52caae9737c8458a9ba643f1593f1e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f522c85df4244eeb87938b7f79305a52"
I20260928 14:59:34.159055 13555 raft_consensus.cc:3037] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:59:34.159725 13555 raft_consensus.cc:2454] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52caae9737c8458a9ba643f1593f1e22 in term 1.
I20260928 14:59:34.159873 13632 leader_election.cc:302] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52caae9737c8458a9ba643f1593f1e22, e7a2b54943754b3b9158a78cbf972587; no voters:
I20260928 14:59:34.160063 13770 raft_consensus.cc:2781] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:59:34.160218 13770 raft_consensus.cc:686] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 1 LEADER]: Becoming Leader. State: Replica: 52caae9737c8458a9ba643f1593f1e22, State: Running, Role: LEADER
I20260928 14:59:34.160311 13770 consensus_queue.cc:237] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } }
I20260928 14:59:34.162156 13255 catalog_manager.cc:5495] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52caae9737c8458a9ba643f1593f1e22 (127.1.252.195). New cstate: current_term: 1 leader_uuid: "52caae9737c8458a9ba643f1593f1e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52caae9737c8458a9ba643f1593f1e22" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 } health_report { overall_health: UNKNOWN } } }
I20260928 14:59:34.223420 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:34.218876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.1.252.195:42103)
0928 14:59:34.218920 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:59:34.218922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:59:34.218932 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:59:34.219038 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:59:34.219039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:59:34.219060 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:59:34.219132 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:59:34.219320 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.220123 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.222229 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:59:34.222551 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:59:34.222621 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:59:34.222865 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:59:34.223017 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:59:34.223094 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:59:34.223305 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 14:59:34.223328 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:59:34.227244 13555 raft_consensus.cc:1261] T 6cad1595615d4f2597ca9fe9dcbadfd4 P f522c85df4244eeb87938b7f79305a52 [term 1 FOLLOWER]: Refusing update from remote peer 52caae9737c8458a9ba643f1593f1e22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:34.227435 13405 raft_consensus.cc:1261] T 6cad1595615d4f2597ca9fe9dcbadfd4 P e7a2b54943754b3b9158a78cbf972587 [term 1 FOLLOWER]: Refusing update from remote peer 52caae9737c8458a9ba643f1593f1e22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:59:34.227797 13774 consensus_queue.cc:1035] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a2b54943754b3b9158a78cbf972587" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:34.227996 13774 consensus_queue.cc:1035] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f522c85df4244eeb87938b7f79305a52" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:59:34.232946 13778 mvcc.cc:205] Tried to move back new op lower bound from 7334328624032108544 to 7334328623761096704. Current Snapshot: MvccSnapshot[applied={T|T < 7334328624032108544}]
I20260928 14:59:34.236074 13779 mvcc.cc:205] Tried to move back new op lower bound from 7334328624032108544 to 7334328623761096704. Current Snapshot: MvccSnapshot[applied={T|T < 7334328624032108544}]
I20260928 14:59:34.236552 13781 mvcc.cc:205] Tried to move back new op lower bound from 7334328624032108544 to 7334328623761096704. Current Snapshot: MvccSnapshot[applied={T|T < 7334328624032108544}]
I20260928 14:59:34.243454 13254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:34.243569 13254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:59:34.244513 13254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca793cc62ebe48e4bee277a0b590c5ee: Sending DeleteTablet for 3 replicas of tablet 6cad1595615d4f2597ca9fe9dcbadfd4
I20260928 14:59:34.245005 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cad1595615d4f2597ca9fe9dcbadfd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:34 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:46928
I20260928 14:59:34.245051 13531 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cad1595615d4f2597ca9fe9dcbadfd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:34 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:47338
I20260928 14:59:34.245787 13676 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cad1595615d4f2597ca9fe9dcbadfd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:59:34 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:50566
I20260928 14:59:34.249046 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 13338
W20260928 14:59:34.261590 13240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:45243 (error 108)
I20260928 14:59:34.262276 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 13476
I20260928 14:59:34.262948 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:59:34.262431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:45243 (local address 127.0.0.1:46934)
0928 14:59:34.262850 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:34.262905 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394}
W20260928 14:59:34.264663 13240 catalog_manager.cc:4584] TS e7a2b54943754b3b9158a78cbf972587 (127.1.252.193:45243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cad1595615d4f2597ca9fe9dcbadfd4: Network error: Client connection negotiation failed: client connection to 127.1.252.193:45243: connect: Connection refused (error 111)
I20260928 14:59:34.265825 13789 tablet_replica.cc:330] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: stopping tablet replica
I20260928 14:59:34.265983 13789 raft_consensus.cc:2229] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:59:34.268824 13789 raft_consensus.cc:2258] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:59:34.269366 13789 ts_tablet_manager.cc:1905] T 6cad1595615d4f2597ca9fe9dcbadfd4 P 52caae9737c8458a9ba643f1593f1e22: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:59:34.273340 13757 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:59:34.273065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36963 (local address 127.0.0.1:47350)
0928 14:59:34.273110 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:34.273121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:34.273157 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:34.273291 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.194:36963: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 14:59:34.276199 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 13614
W20260928 14:59:34.282106 13757 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:59:34.281819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:42103 (local address 127.0.0.1:50588)
0928 14:59:34.281862 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:59:34.281873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:59:34.281910 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:59:34.282066 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.195:42103: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 14:59:34.282529 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 13223
I20260928 14:59:34.288884 2035 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:59:34.633601 2035 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:59:34.633723 2035 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:59:34.978508 2035 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:59:34.980 UTC [12808] LOG: received smart shutdown request
2026-09-28 14:59:34.984 UTC [12808] LOG: background worker "logical replication launcher" (PID 12817) exited with exit code 1
2026-09-28 14:59:34.984 UTC [12812] LOG: shutting down
2026-09-28 14:59:35.179 UTC [12808] LOG: database system is shut down
2026-09-28T14:59:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46915 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:59:35 dist-test-slave-sf70 krb5kdc[13799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:59:35 dist-test-slave-sf70 krb5kdc[13799](info): set up 2 sockets
Sep 28 14:59:35 dist-test-slave-sf70 krb5kdc[13799](info): commencing operation
krb5kdc: starting...
W20260928 14:59:36.321521 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 14:59:36 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:59:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/rangeradmin_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/rangeradmin_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/rangerlookup_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/rangerlookup_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
I20260928 14:59:36.482084 2035 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWmXXpN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/postgres -l logfile start
2026-09-28 14:59:39.109 UTC [13829] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:59:39.109 UTC [13829] LOG: listening on IPv4 address "127.1.252.212", port 40061
2026-09-28 14:59:39.113 UTC [13829] LOG: listening on Unix socket "/tmp/.s.PGSQL.40061"
2026-09-28 14:59:39.135 UTC [13832] LOG: database system was shut down at 2026-09-28 14:59:37 UTC
2026-09-28 14:59:39.138 UTC [13829] LOG: database system is ready to accept connections
I20260928 14:59:40.089442 2035 mini_postgres.cc:96] Postgres bound to 40061
2026-09-28 14:59:40.095 UTC [13840] FATAL: database "slave" does not exist
127.1.252.212:40061 - accepting connections
I20260928 14:59:40.096092 2035 mini_ranger.cc:161] Starting Ranger...
I20260928 14:59:40.110424 2035 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:59:40.664373 2035 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:59:40.664474 2035 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-admin
2026-09-28 14:59:40,966 [I] DB FLAVOR :POSTGRES
2026-09-28 14:59:40,966 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:59:40,966 [I] Checking connection..
2026-09-28 14:59:40,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:59:41,212 [I] Checking connection passed.
2026-09-28 14:59:41,213 [I] --------- Verifying version history table ---------
2026-09-28 14:59:41,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:59:41,451 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:59:41,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:59:41,720 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:59:41,720 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:59:41,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:59:41.946 UTC [13904] WARNING: there is no transaction in progress
2026-09-28 14:59:41,952 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:59:41,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:59:42,214 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:59:42,214 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:59:42,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:59:42,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:59:42,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sf70.c.gcp-upstream.internal','N') ;"
2026-09-28 14:59:42,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:59:43,256 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:59:43,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:59:43,518 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:59:43,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:59:43,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:59:44,042 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:59:44,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:59:45.050 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.069 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.087 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.103 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.124 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.142 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.197 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.205 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.219 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.230 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.240 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.255 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.265 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.271 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.279 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45.285 UTC [14012] WARNING: there is no transaction in progress
2026-09-28 14:59:45,963 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:59:45,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:59:46,212 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:59:46,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:59:46,454 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:59:46,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:59:46,713 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:59:46,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:59:46,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:59:47,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sf70.c.gcp-upstream.internal';"
2026-09-28 14:59:47,456 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:59:47,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:59:47,710 [I] DB_PATCHES have already been applied
I20260928 14:59:47.715618 2035 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWmXXpN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:59:47.715685 2035 mini_ranger.cc:193] Using host: 127.1.252.212
I20260928 14:59:47.718765 2035 mini_ranger.cc:227] Ranger admin URL: http://127.1.252.212:58573
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:59:48.823505 2035 mini_ranger.cc:160] Time spent starting Ranger: real 8.727s user 0.001s sys 0.007s
Sep 28, 2026 3:00:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:00:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 3:00:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:00:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:00:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:00:10.361784 2035 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.252.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.252.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/rangerkms_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.252.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/rangerkms_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.252.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.252.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.252.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/HTTP_127.1.252.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:00:10 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607610, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:00:10.463380 2035 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:00:10.472939 2035 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:00:11.430883 2035 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:00:11.430974 2035 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms
I20260928 15:00:11.430995 2035 mini_ranger_kms.cc:227] Using postgres at 127.1.252.212:40061
2026-09-28 15:00:11,596 [I] DB FLAVOR :POSTGRES
2026-09-28 15:00:11,596 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:00:11,596 [I] Checking connection
2026-09-28 15:00:11,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:00:11,824 [I] connection success
2026-09-28 15:00:11,824 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:00:11,825 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:00:11,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:00:12,079 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:00:12,080 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:00:12,080 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:00:12,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.252.212:40061/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:00:12,343 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:00:12.789908 2035 mini_ranger_kms.cc:310] Created kms service
I20260928 15:00:12.975950 2035 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:00:13.020164 2035 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:00:13 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607610, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:13.405658 2035 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:00:13.405893 2035 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWmXXpN/build/release/bin/postgresql.jar:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWmXXpN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:00:13.405918 2035 mini_ranger_kms.cc:240] Using host: 127.1.252.212
I20260928 15:00:13.409859 2035 mini_ranger_kms.cc:276] Ranger KMS PID: 14192
I20260928 15:00:13.409935 2035 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.252.212:57643
Sep 28, 2026 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:00:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:00:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:00:15.683451 2035 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.220s user 0.002s sys 0.146s
I20260928 15:00:15.683558 2035 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:00:15.683604 2035 mini_ranger_kms.cc:401] 127.1.252.212:57643/kms/v1/keys
Sep 28 15:00:15 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.252.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:00:16 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.252.212@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
Sep 28 15:00:16 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.252.212@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
Sep 28 15:00:18 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:00:18.936141 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37981
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:37981
--ranger_config_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:19.017040 14243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:19.017259 14243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:19.017294 14243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:19.018410 14243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:19.018477 14243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:19.018504 14243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:19.018532 14243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:19.018564 14243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:19.020167 14243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.252.212:57643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:37981
--ranger_config_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:19.020417 14243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:19.020653 14243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:19.025561 14249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:19.025628 14251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:19.025794 14248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:19.025811 14243 server_base.cc:1034] running on GCE node
I20260928 15:00:19.026070 14243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:19.026412 14243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:19.027590 14243 hybrid_clock.cc:648] HybridClock initialized: now 1790607619027581 us; error 29 us; skew 500 ppm
Sep 28 15:00:19 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:19.030628 14243 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:19.031109 14243 webserver.cc:460] Webserver started at http://127.1.252.254:37331/ using document root <none> and password file <none>
I20260928 15:00:19.031392 14243 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:19.031435 14243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:19.031576 14243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:00:19 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:19.090955 14243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3970a5320c0a1c2b5760a032dd2622b"
server_key_iv: "c2be534f0b9b292dc0268ac85756fb20"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:19.091374 14243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3970a5320c0a1c2b5760a032dd2622b"
server_key_iv: "c2be534f0b9b292dc0268ac85756fb20"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:19.092913 14243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:00:19.108470 14258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:19.108645 14243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:19.108733 14243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3970a5320c0a1c2b5760a032dd2622b"
server_key_iv: "c2be534f0b9b292dc0268ac85756fb20"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:19.108831 14243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:19.129140 14243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:19.131258 14243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:19.131410 14243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:19.136474 14243 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:37981
I20260928 15:00:19.136552 14310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:37981 every 8 connection(s)
I20260928 15:00:19.136905 14243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:19.137669 14311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:19.140681 14311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Bootstrap starting.
I20260928 15:00:19.141362 14311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:19.141670 14311 log.cc:824] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:19.142465 14311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: No bootstrap required, opened a new log
I20260928 15:00:19.142547 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14243
I20260928 15:00:19.142658 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:00:19.142844 2035 external_mini_cluster.cc:1428] Setting key d9bd20790aea8be89f5c202907f84801
I20260928 15:00:19.145222 14311 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:19.145421 14311 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:19.145485 14311 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dd6c88a5ff49759f5af59fa8ce4a2c, State: Initialized, Role: FOLLOWER
I20260928 15:00:19.145635 14311 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:19.145735 14311 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:19.145780 14311 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:19.145828 14311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:19.146498 14311 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:19.146602 14311 leader_election.cc:302] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9dd6c88a5ff49759f5af59fa8ce4a2c; no voters:
I20260928 15:00:19.146786 14311 leader_election.cc:288] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:00:19.146836 14316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:19.147048 14316 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 LEADER]: Becoming Leader. State: Replica: e9dd6c88a5ff49759f5af59fa8ce4a2c, State: Running, Role: LEADER
I20260928 15:00:19.147126 14311 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:19.147193 14316 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:19.147564 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } } }
I20260928 15:00:19.147701 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:19.147872 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dd6c88a5ff49759f5af59fa8ce4a2c. Latest consensus state: current_term: 1 leader_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } } }
I20260928 15:00:19.148008 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:19.148162 14326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:19.148670 14326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:19.148938 14311 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:00:19.150364 14334 mvcc.cc:205] Tried to move back new op lower bound from 7334328808038031360 to 7334328808027459584. Current Snapshot: MvccSnapshot[applied={T|T < 7334328808038031360}]
Sep 28 15:00:19 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:19.155869 14326 catalog_manager.cc:1347] Generated new cluster ID: 599a7964316944bfbee8fb3499d26e17
I20260928 15:00:19.155979 14326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:19.157795 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:19.143352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.1.252.254:37981)
0928 15:00:19.143543 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:19.143553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:19.143645 (+ 92us) server_negotiation.cc:407] Connection header received
0928 15:00:19.145203 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:19.145210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:19.145260 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:19.145527 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:19.146165 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:19.147442 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:19.148594 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:19.149160 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:19.152453 (+ 3293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:19.152893 (+ 440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:19.152897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:19.152897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:19.152907 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:19.155904 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:19.156325 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:19.156328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:19.156328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:19.156424 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:19.156613 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:19.156613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:19.156613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:19.156885 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:19.157396 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:19.157460 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 15:00:19.157602 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 15:00:19.174877 14326 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:00:19.175787 14326 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:19.184046 14326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Generated new TSK 0
I20260928 15:00:19.184458 14326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 15:00:19.848184 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37981
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:19.937652 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:19.937883 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:19.937922 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:19.939128 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:19.939198 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:19.939244 14357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:00:19.940778 14357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.252.212:57643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:19.941044 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:19.941286 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:19.946296 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:19.946413 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:19.946761 14357 server_base.cc:1034] running on GCE node
W20260928 15:00:19.946810 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:19.947041 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:19.947314 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:19.948463 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790607619948435 us; error 46 us; skew 500 ppm
Sep 28 15:00:19 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:19.951256 14357 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:00:19.951682 14357 webserver.cc:460] Webserver started at http://127.1.252.193:44073/ using document root <none> and password file <none>
I20260928 15:00:19.951998 14357 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:19.952070 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:19.952204 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:00:19 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:19.965873 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "9e17c9ff2fe44b9db40fd6ee5a307755"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3cd1199b21c2e7c2fdcf824cf0d96ac"
server_key_iv: "988ef0d6842c2603203a86665ee00b19"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:19.966429 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "9e17c9ff2fe44b9db40fd6ee5a307755"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3cd1199b21c2e7c2fdcf824cf0d96ac"
server_key_iv: "988ef0d6842c2603203a86665ee00b19"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:19.969281 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:00:19.982407 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:19.982613 14357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:19.982712 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "9e17c9ff2fe44b9db40fd6ee5a307755"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3cd1199b21c2e7c2fdcf824cf0d96ac"
server_key_iv: "988ef0d6842c2603203a86665ee00b19"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:19.982821 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:19.999639 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:20.001607 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:20.001785 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:20.002058 14357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:20.002408 14357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:20.002471 14357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.002593 14357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:20.002647 14357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.010916 14357 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:34513
I20260928 15:00:20.011006 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:34513 every 8 connection(s)
I20260928 15:00:20.011324 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:20.013680 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14357
I20260928 15:00:20.013790 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:00:20.013972 2035 external_mini_cluster.cc:1428] Setting key d9e73bb39836045605f6d20ee527bc86
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 15:00:20.023242 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.012177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37981 (local address 127.1.252.193:55959)
0928 15:00:20.012373 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:20.012421 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:20.013097 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:20.013376 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:20.013380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:20.013634 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:20.013942 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.013949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.014996 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.014998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.016354 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.016358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.016817 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.016818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.016951 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.017528 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:20.017538 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:20.019308 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:20.021434 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.021441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.021444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.021776 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.022044 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.022045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.022046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.022192 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.022514 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:20.022521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:20.022696 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 15:00:20.022808 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 15:00:20.022981 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260928 15:00:20.023571 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.012226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:55959 (local address 127.1.252.254:37981)
0928 15:00:20.012385 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:20.012387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:20.012517 (+ 130us) server_negotiation.cc:407] Connection header received
0928 15:00:20.013207 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:20.013208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:20.013223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:20.013299 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:20.014065 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.014889 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.016455 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.016715 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.016799 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.019426 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:20.019427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:20.019427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:20.019435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:20.021335 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.021885 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.021886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.021886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.021957 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.022284 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.022285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.022285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.022434 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:20.022506 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:20.023395 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 15:00:20.023485 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260928 15:00:20.023958 14486 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37981
I20260928 15:00:20.024077 14486 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:20.024268 14486 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:20.024817 14275 ts_manager.cc:194] Registered new tserver with Master: 9e17c9ff2fe44b9db40fd6ee5a307755 (127.1.252.193:34513)
I20260928 15:00:20.025528 14275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:55959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 15:00:20.042508 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37981
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:20.154080 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:20.154350 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:20.154386 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:20.156060 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:20.156117 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:20.156157 14494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:00:20.158334 14494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.252.212:57643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:20.158618 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:20.158905 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:20.165361 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.165434 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.165601 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:20.166046 14494 server_base.cc:1034] running on GCE node
I20260928 15:00:20.166263 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:20.166563 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:20.167742 14494 hybrid_clock.cc:648] HybridClock initialized: now 1790607620167723 us; error 39 us; skew 500 ppm
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:20.171197 14494 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:00:20.171764 14494 webserver.cc:460] Webserver started at http://127.1.252.194:41601/ using document root <none> and password file <none>
I20260928 15:00:20.172075 14494 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:20.172140 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:20.172302 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:20.188463 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "356500cf902c48a59d97486b801499ce"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "0314c8c5ddf4a8d8268b8d1a2aa1d788"
server_key_iv: "a003d687d5ff046cf18e41ca3bf5d333"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.188912 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "356500cf902c48a59d97486b801499ce"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "0314c8c5ddf4a8d8268b8d1a2aa1d788"
server_key_iv: "a003d687d5ff046cf18e41ca3bf5d333"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.190836 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:00:20.208076 14511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.208359 14494 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 15:00:20.208467 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "356500cf902c48a59d97486b801499ce"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "0314c8c5ddf4a8d8268b8d1a2aa1d788"
server_key_iv: "a003d687d5ff046cf18e41ca3bf5d333"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.208580 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:20.231711 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:20.234405 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:20.234642 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:20.234926 14494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:20.235481 14494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:20.235579 14494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.235672 14494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:20.235737 14494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.242467 14494 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42391
I20260928 15:00:20.242982 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:20.251461 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42391 every 8 connection(s)
I20260928 15:00:20.253008 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14494
I20260928 15:00:20.253118 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:00:20.253332 2035 external_mini_cluster.cc:1428] Setting key 293ee2eff7de82f20ca1a730008bfda2
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:20.265853 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.252925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:55545 (local address 127.1.252.254:37981)
0928 15:00:20.252971 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:20.252973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:20.252986 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:00:20.256092 (+ 3106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:20.256094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:20.256110 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:20.256139 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:20.257003 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.257791 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.259363 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.259707 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.259751 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.262556 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:20.262558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:20.262558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:20.262566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:20.264302 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.264794 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.264795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.264796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.264855 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.265124 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.265125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.265126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.265253 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:20.265294 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:20.265663 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 15:00:20.265752 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:00:20.266619 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.252579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37981 (local address 127.1.252.194:55545)
0928 15:00:20.252772 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:20.252816 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:20.255964 (+ 3148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:20.256335 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:20.256339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:20.256597 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:20.256889 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.256895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.257911 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.257912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.259262 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.259265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.259859 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.259861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.260037 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.260723 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:20.260733 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:20.262449 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:20.264395 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.264401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.264403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.264694 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.264931 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.264932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.264932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.265055 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.265338 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:20.265344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:20.265501 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 15:00:20.266221 (+ 720us) client_negotiation.cc:240] Negotiation successful
0928 15:00:20.266385 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":125,"threads_started":1}
I20260928 15:00:20.267297 14625 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37981
I20260928 15:00:20.267396 14625 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:20.267591 14625 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:20.267994 14275 ts_manager.cc:194] Registered new tserver with Master: 356500cf902c48a59d97486b801499ce (127.1.252.194:42391)
I20260928 15:00:20.268465 14275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:55545
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 15:00:20.298906 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37981
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:20.424741 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:20.425072 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:20.425155 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:20.426816 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:20.426913 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:20.427008 14631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:00:20.429466 14631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.252.212:57643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:20.429916 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:20.430243 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:20.436636 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.436998 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.437695 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:20.438241 14631 server_base.cc:1034] running on GCE node
I20260928 15:00:20.438630 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:20.438884 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:20.440035 14631 hybrid_clock.cc:648] HybridClock initialized: now 1790607620440013 us; error 34 us; skew 500 ppm
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:20.443300 14631 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:00:20.443799 14631 webserver.cc:460] Webserver started at http://127.1.252.195:42237/ using document root <none> and password file <none>
I20260928 15:00:20.444092 14631 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:20.444144 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:20.444264 14631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:20.462625 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "bd16cb438f7046fb83a37710f5ec57fc"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "d8cc0c728c75073a153a572856df806f"
server_key_iv: "57fe2958615952b0e296332e6df63b6f"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.463150 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "bd16cb438f7046fb83a37710f5ec57fc"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "d8cc0c728c75073a153a572856df806f"
server_key_iv: "57fe2958615952b0e296332e6df63b6f"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.465847 14631 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:00:20.485183 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.485419 14631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:00:20.485505 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "bd16cb438f7046fb83a37710f5ec57fc"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "d8cc0c728c75073a153a572856df806f"
server_key_iv: "57fe2958615952b0e296332e6df63b6f"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.485585 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:20.501959 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:20.504315 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:20.504474 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:20.504734 14631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:20.505095 14631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:20.505148 14631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.505216 14631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:20.505244 14631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.512740 14631 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:42885
I20260928 15:00:20.512780 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:42885 every 8 connection(s)
I20260928 15:00:20.513149 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:20.522888 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14631
I20260928 15:00:20.522993 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:00:20.523162 2035 external_mini_cluster.cc:1428] Setting key f2e62658a65f2d103f107d027cf5aa45
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:20.542160 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.513900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37981 (local address 127.1.252.195:50277)
0928 15:00:20.514053 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:20.516014 (+ 1961us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:20.516681 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:20.517025 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:20.517029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:20.517261 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:20.517531 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.517537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.518505 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.518506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.519731 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.519734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.520286 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.520287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.520422 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.536073 (+ 15651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:20.536086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:20.537944 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:20.540000 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.540009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.540012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.540546 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.540852 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.540853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.540853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.541082 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.541447 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:20.541454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:20.541625 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 15:00:20.541739 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:00:20.541910 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 15:00:20.542754 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.513879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:50277 (local address 127.1.252.254:37981)
0928 15:00:20.513931 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:20.513933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:20.516113 (+ 2180us) server_negotiation.cc:407] Connection header received
0928 15:00:20.516802 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:20.516804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:20.516817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:20.516847 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:20.517638 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.518393 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.519831 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.520144 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.520184 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.538075 (+ 17891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:20.538077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:20.538077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:20.538087 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:20.539925 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.540650 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.540652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.540652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.540745 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.541169 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.541178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.541178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.541348 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:20.541396 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:20.542573 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0928 15:00:20.542662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:00:20.543089 14760 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37981
I20260928 15:00:20.543192 14760 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:20.543381 14760 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:20.543735 14275 ts_manager.cc:194] Registered new tserver with Master: bd16cb438f7046fb83a37710f5ec57fc (127.1.252.195:42885)
I20260928 15:00:20.544260 14275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:50277
I20260928 15:00:20.548273 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:20.548929 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14357
I20260928 15:00:20.557579 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14494
I20260928 15:00:20.563670 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14631
I20260928 15:00:20.571672 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14243
I20260928 15:00:20.577540 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37981
--webserver_interface=127.1.252.254
--webserver_port=37331
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:37981
--ranger_config_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:20.661953 14763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:20.662161 14763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:20.662182 14763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:20.663319 14763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:20.663364 14763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:20.663378 14763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:20.663391 14763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:20.663403 14763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:20.664961 14763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:37981
--ranger_config_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:20.665201 14763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:20.665443 14763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:20.671161 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.671166 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:20.671437 14763 server_base.cc:1034] running on GCE node
W20260928 15:00:20.671582 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:20.671877 14763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:20.672235 14763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:20.673394 14763 hybrid_clock.cc:648] HybridClock initialized: now 1790607620673373 us; error 39 us; skew 500 ppm
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:20.676429 14763 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:20.676908 14763 webserver.cc:460] Webserver started at http://127.1.252.254:37331/ using document root <none> and password file <none>
I20260928 15:00:20.677145 14763 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:20.677207 14763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:20.727815 14763 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:20.728611 14779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.728785 14763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:00:20.728896 14763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3970a5320c0a1c2b5760a032dd2622b"
server_key_iv: "c2be534f0b9b292dc0268ac85756fb20"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.729274 14763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:20.746214 14763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:20.747780 14763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:20.747958 14763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:20.752115 14763 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:37981
I20260928 15:00:20.752158 14831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:37981 every 8 connection(s)
I20260928 15:00:20.752609 14763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:20.753794 14832 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:00:20.753827 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14763
I20260928 15:00:20.754106 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:34513
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:37981
--webserver_port=44073
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:20.754519 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Bootstrap starting.
I20260928 15:00:20.762758 14832 log.cc:824] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:20.764473 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:20.764763 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Bootstrap complete.
I20260928 15:00:20.767153 14832 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:20.767405 14832 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dd6c88a5ff49759f5af59fa8ce4a2c, State: Initialized, Role: FOLLOWER
I20260928 15:00:20.767583 14832 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:20.767663 14832 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:20.767714 14832 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:20.767761 14832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:00:20.768790 14832 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:20.768908 14832 leader_election.cc:302] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9dd6c88a5ff49759f5af59fa8ce4a2c; no voters:
I20260928 15:00:20.769093 14832 leader_election.cc:288] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:00:20.769129 14836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:00:20.769237 14836 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [term 2 LEADER]: Becoming Leader. State: Replica: e9dd6c88a5ff49759f5af59fa8ce4a2c, State: Running, Role: LEADER
I20260928 15:00:20.769364 14832 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:20.769398 14836 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } }
I20260928 15:00:20.769749 14838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dd6c88a5ff49759f5af59fa8ce4a2c. Latest consensus state: current_term: 2 leader_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } } }
I20260928 15:00:20.769832 14838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:20.770046 14837 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dd6c88a5ff49759f5af59fa8ce4a2c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37981 } } }
I20260928 15:00:20.770125 14837 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:20.770844 14832 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:00:20.773216 14842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:20.774348 14842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:20.774626 14842 catalog_manager.cc:1259] Loaded cluster ID: 599a7964316944bfbee8fb3499d26e17
I20260928 15:00:20.774761 14842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:20.776150 14842 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:20.776358 14842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Loaded TSK: 0
I20260928 15:00:20.776603 14842 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:00:20.864338 14834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:20.864602 14834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:20.864639 14834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:20.866452 14834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:20.866533 14834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:20.866588 14834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:00:20.868750 14834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:34513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=44073
--tserver_master_addrs=127.1.252.254:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:20.868975 14834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:20.869251 14834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:20.876197 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.876561 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:20.876684 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:20.877012 14834 server_base.cc:1034] running on GCE node
I20260928 15:00:20.877239 14834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:20.877535 14834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:20.881263 14834 hybrid_clock.cc:648] HybridClock initialized: now 1790607620880598 us; error 673 us; skew 500 ppm
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:20.884519 14834 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:00:20.885071 14834 webserver.cc:460] Webserver started at http://127.1.252.193:44073/ using document root <none> and password file <none>
I20260928 15:00:20.885362 14834 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:20.885444 14834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:20.908876 14834 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 15:00:20.910663 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.910876 14834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 15:00:20.910969 14834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "9e17c9ff2fe44b9db40fd6ee5a307755"
format_stamp: "Formatted at 2026-09-28 15:00:19 on dist-test-slave-sf70"
server_key: "f3cd1199b21c2e7c2fdcf824cf0d96ac"
server_key_iv: "988ef0d6842c2603203a86665ee00b19"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:20.911412 14834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:20.940599 14834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:20.943075 14834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:20.943238 14834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:20.943465 14834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:20.943831 14834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:20.943879 14834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.943951 14834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:20.943976 14834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:20.951252 14834 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:34513
I20260928 15:00:20.951364 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:34513 every 8 connection(s)
I20260928 15:00:20.951743 14834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 15:00:20 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:20.960264 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14834
I20260928 15:00:20.960551 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:42391
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:37981
--webserver_port=41601
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:20.963265 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.952850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:53909 (local address 127.1.252.254:37981)
0928 15:00:20.953047 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:20.953055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:20.953222 (+ 167us) server_negotiation.cc:407] Connection header received
0928 15:00:20.953937 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:20.953942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:20.953982 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:20.954086 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:20.954781 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.955645 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.957084 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.957552 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.957633 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.959839 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:20.959843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:20.959843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:20.959852 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:20.961572 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.961998 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.962001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.962001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.962088 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:20.962362 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:20.962363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:20.962363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:20.962579 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:20.962674 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:20.962956 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 15:00:20.963057 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260928 15:00:20.963276 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:20.952746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37981 (local address 127.1.252.193:53909)
0928 15:00:20.953065 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:20.953109 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:20.953744 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:20.954170 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:20.954174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:20.954395 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:20.954672 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.954678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.955733 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.955734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.956992 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:20.956996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:20.957674 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:20.957676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:20.957816 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:20.958303 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:20.958313 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:20.959733 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:20.961670 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.961676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.961678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.961899 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.962157 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:20.962158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:20.962159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:20.962288 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:20.962655 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:20.962661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:20.962796 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 15:00:20.962905 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:00:20.963055 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":81,"threads_started":1}
I20260928 15:00:20.964550 14994 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37981
I20260928 15:00:20.964646 14994 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:20.964826 14994 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
W20260928 15:00:20.965685 14994 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:37981 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:00:20.965770 14994 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:20.965835 14994 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:20.966184 14994 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:20.966360 14994 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
W20260928 15:00:20.966642 14994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:00:21.079563 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:21.079842 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:21.079874 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:21.081502 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:21.081561 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:21.081604 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:00:21.083777 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:42391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=41601
--tserver_master_addrs=127.1.252.254:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:21.084055 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:21.084304 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:21.090107 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:21.090111 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:21.090121 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:21.090953 14998 server_base.cc:1034] running on GCE node
I20260928 15:00:21.091168 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:21.091429 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:21.092610 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790607621092565 us; error 56 us; skew 500 ppm
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:21.095621 14998 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:00:21.096139 14998 webserver.cc:460] Webserver started at http://127.1.252.194:41601/ using document root <none> and password file <none>
I20260928 15:00:21.096362 14998 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:21.096443 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:21.115933 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 15:00:21.117014 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:21.117228 14998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:21.117312 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "356500cf902c48a59d97486b801499ce"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "0314c8c5ddf4a8d8268b8d1a2aa1d788"
server_key_iv: "a003d687d5ff046cf18e41ca3bf5d333"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:21.117738 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:21.143988 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:21.146399 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:21.146561 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:21.146827 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:21.147149 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:21.147204 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:21.147248 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:21.147266 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:21.154454 14998 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42391
I20260928 15:00:21.154564 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42391 every 8 connection(s)
I20260928 15:00:21.154922 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:21.157694 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 14998
I20260928 15:00:21.157995 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.252.212:57643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:42885
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:37981
--webserver_port=42237
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:21.167303 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:21.155746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:51991 (local address 127.1.252.254:37981)
0928 15:00:21.155800 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:21.155803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:21.156040 (+ 237us) server_negotiation.cc:407] Connection header received
0928 15:00:21.156716 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:21.156718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:21.156731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:21.156785 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:21.157463 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.158161 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.160903 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.161225 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.161299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:21.163876 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:21.163877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:21.163878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:21.163924 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:21.165663 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:21.166175 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:21.166176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:21.166177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:21.166240 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:21.166533 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:21.166534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:21.166535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:21.166702 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:21.166778 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:21.167109 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 15:00:21.167202 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:00:21.167449 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:21.155696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37981 (local address 127.1.252.194:51991)
0928 15:00:21.155913 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:21.155960 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:21.156589 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:21.156858 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:21.156863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:21.157088 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:21.157362 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:21.157368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.159485 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.159486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:21.160801 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:21.160805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.161355 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.161357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:21.161497 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:21.162091 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:21.162107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:21.163772 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:21.165758 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:21.165764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:21.165766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:21.166058 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:21.166316 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:21.166317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:21.166317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:21.166447 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:21.166786 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:21.166792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:21.166949 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 15:00:21.167068 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:00:21.167225 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260928 15:00:21.168017 15128 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37981
I20260928 15:00:21.168121 15128 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.168315 15128 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
W20260928 15:00:21.169018 15128 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:37981 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:00:21.169087 15128 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.169144 15128 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:21.169499 15128 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.169569 15128 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
W20260928 15:00:21.169991 15128 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:00:21.274725 15131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:21.275064 15131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:21.275152 15131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:21.276913 15131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:21.277015 15131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:21.277096 15131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:00:21.279191 15131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.252.212:57643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:42885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=42237
--tserver_master_addrs=127.1.252.254:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:21.279630 15131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:21.279950 15131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:21.287535 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:21.287655 15140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:21.287820 15137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:21.293924 15131 server_base.cc:1034] running on GCE node
I20260928 15:00:21.294162 15131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:21.294490 15131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:21.295734 15131 hybrid_clock.cc:648] HybridClock initialized: now 1790607621295711 us; error 37 us; skew 500 ppm
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:21.298908 15131 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:00:21.299486 15131 webserver.cc:460] Webserver started at http://127.1.252.195:42237/ using document root <none> and password file <none>
I20260928 15:00:21.299731 15131 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:21.299798 15131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:21.318018 15131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:21.319044 15148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:21.319218 15131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:21.319301 15131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "bd16cb438f7046fb83a37710f5ec57fc"
format_stamp: "Formatted at 2026-09-28 15:00:20 on dist-test-slave-sf70"
server_key: "d8cc0c728c75073a153a572856df806f"
server_key_iv: "57fe2958615952b0e296332e6df63b6f"
server_key_version: "kuduclusterkey@0"
I20260928 15:00:21.319768 15131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:21.352164 15131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:21.354527 15131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:21.354686 15131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:21.354946 15131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:21.355311 15131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:21.355361 15131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:21.355393 15131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:21.355410 15131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:21.362715 15131 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:42885
I20260928 15:00:21.362809 15261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:42885 every 8 connection(s)
I20260928 15:00:21.363240 15131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:21.364914 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15131
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:21.374859 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:21.364021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37981 (local address 127.1.252.195:37153)
0928 15:00:21.364180 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:21.364231 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:21.364934 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:21.365371 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:21.365375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:21.365591 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:21.365851 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:21.365856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.366846 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.366847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:21.368251 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:21.368253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.368841 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.368842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:21.368980 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:21.369590 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:21.369607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:21.371106 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:21.373255 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:21.373261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:21.373263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:21.373561 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:21.373782 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:21.373783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:21.373784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:21.373891 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:21.374159 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:21.374165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:21.374329 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 15:00:21.374442 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 15:00:21.374630 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 15:00:21.375425 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:21.364009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:37153 (local address 127.1.252.254:37981)
0928 15:00:21.364069 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:21.364072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:21.364332 (+ 260us) server_negotiation.cc:407] Connection header received
0928 15:00:21.365143 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:21.365144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:21.365157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:21.365189 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:21.365963 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.366731 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.368368 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.368675 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.368726 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:21.371214 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:21.371216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:21.371216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:21.371224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:21.373172 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:21.373645 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:21.373646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:21.373646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:21.373709 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:21.373963 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:21.373964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:21.373964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:21.374081 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:21.374157 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:21.375236 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0928 15:00:21.375301 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 15:00:21.375862 15262 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37981
I20260928 15:00:21.376006 15262 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.376187 15262 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
W20260928 15:00:21.376844 15262 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:37981 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:00:21.376929 15262 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.376997 15262 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:21.377327 15262 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.377408 15262 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
W20260928 15:00:21.378232 15262 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:00:21.379163 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:21.371669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.1.252.254:37981)
0928 15:00:21.371817 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:21.371819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:21.372079 (+ 260us) server_negotiation.cc:407] Connection header received
0928 15:00:21.372192 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:21.372193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:21.372205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:21.372247 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:21.373069 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.373663 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.374699 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:21.375003 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:21.375088 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:21.376516 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:21.376518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:21.376519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:21.376526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:21.378058 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:21.378446 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:21.378447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:21.378447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:21.378507 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:21.378731 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:21.378731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:21.378732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:21.378830 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:21.378940 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:21.379037 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 15:00:21.379097 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
Sep 28 15:00:21 dist-test-slave-sf70 krb5kdc[13799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:21.966848 14994 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:21.967012 14994 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:21.967361 14796 ts_manager.cc:194] Registered new tserver with Master: 9e17c9ff2fe44b9db40fd6ee5a307755 (127.1.252.193:34513)
I20260928 15:00:21.968147 14796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:53909
I20260928 15:00:22.170176 15128 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:22.170331 15128 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:22.170729 14796 ts_manager.cc:194] Registered new tserver with Master: 356500cf902c48a59d97486b801499ce (127.1.252.194:42391)
I20260928 15:00:22.171267 14796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:51991
Sep 28 15:00:22 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
I20260928 15:00:22.378428 15262 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:22.378578 15262 heartbeater.cc:511] Master 127.1.252.254:37981 requested a full tablet report, sending...
I20260928 15:00:22.378887 14796 ts_manager.cc:194] Registered new tserver with Master: bd16cb438f7046fb83a37710f5ec57fc (127.1.252.195:42885)
I20260928 15:00:22.379400 14796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:37153
I20260928 15:00:22.388343 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:22.397426 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.389423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.1.252.254:37981)
0928 15:00:22.389579 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.389581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.389590 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:00:22.389619 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.389620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.389632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.389706 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:22.390342 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.391106 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.392173 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.392468 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.392550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.394129 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:22.394130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:22.394131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:22.394138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:22.395963 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:22.396340 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:22.396341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:22.396341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:22.396395 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:22.396585 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:22.396586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:22.396586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:22.396708 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:22.396801 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.397243 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.397325 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 15:00:22.399900 14796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:00:22.402287 14796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:00:22.412328 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.405238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:34513 (local address 127.0.0.1:38016)
0928 15:00:22.405377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:22.405415 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:22.405478 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:22.408361 (+ 2883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:22.408364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:22.408367 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:22.408438 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.408441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.409695 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.409696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.410967 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.410970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.412039 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.412040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.412119 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.412132 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:00:22.412200 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:00:22.412246 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260928 15:00:22.412765 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.405345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.1.252.193:34513)
0928 15:00:22.408055 (+ 2710us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.408060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.408072 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:00:22.408108 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.408110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.408201 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.408346 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:22.408540 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.409592 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.411083 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.411823 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.411931 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.411992 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.412642 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.412683 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2678,"thread_start_us":50,"threads_started":1}
I20260928 15:00:22.412904 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.405574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.1.252.194:42391)
0928 15:00:22.405727 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.405731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.408640 (+ 2909us) server_negotiation.cc:407] Connection header received
0928 15:00:22.408739 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.408741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.408762 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.408812 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:22.409026 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.410003 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.411567 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.412328 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.412375 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.412423 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.412789 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.412831 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260928 15:00:22.412726 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.405478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42391 (local address 127.0.0.1:41752)
0928 15:00:22.408589 (+ 3111us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:22.408596 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:22.408663 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:22.408914 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:22.408914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:22.408916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:22.408958 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.408960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.410102 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.410103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.411465 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.411468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.412484 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.412485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.412546 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.412549 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:00:22.412614 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:00:22.412651 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3099,"thread_start_us":62,"threads_started":1}
I20260928 15:00:22.413731 14910 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9762a770404cde831dc3e478cf3796 (DEFAULT_TABLE table=test-table [id=5e2d6fcd72c740968e8b55663bb9126f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:22.414099 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9762a770404cde831dc3e478cf3796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:22.414232 15062 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9762a770404cde831dc3e478cf3796 (DEFAULT_TABLE table=test-table [id=5e2d6fcd72c740968e8b55663bb9126f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:22.414561 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9762a770404cde831dc3e478cf3796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:22.417150 15295 tablet_bootstrap.cc:492] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: Bootstrap starting.
I20260928 15:00:22.421247 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.405245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:42885 (local address 127.0.0.1:58810)
0928 15:00:22.408464 (+ 3219us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:22.408474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:22.408516 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:22.417528 (+ 9012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:22.417529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:22.417531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:22.417572 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.417574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.418803 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.418804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.419948 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.419951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.420981 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.420982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.421060 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.421064 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:00:22.421117 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:00:22.421166 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3202,"thread_start_us":87,"threads_started":1}
I20260928 15:00:22.421203 15295 tablet_bootstrap.cc:654] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:22.421480 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.405492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.1.252.195:42885)
0928 15:00:22.413081 (+ 7589us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.413086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.413096 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:00:22.417395 (+ 4299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.417399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.417417 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.417643 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:22.417743 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.418701 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.420056 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.420821 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.420896 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.420945 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.421348 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.421398 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7553,"thread_start_us":59,"threads_started":1}
I20260928 15:00:22.421914 15295 log.cc:824] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:22.421922 15296 tablet_bootstrap.cc:492] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Bootstrap starting.
I20260928 15:00:22.422281 15192 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9762a770404cde831dc3e478cf3796 (DEFAULT_TABLE table=test-table [id=5e2d6fcd72c740968e8b55663bb9126f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:22.422511 15296 tablet_bootstrap.cc:654] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:22.422605 15192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9762a770404cde831dc3e478cf3796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:22.422839 15295 tablet_bootstrap.cc:492] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: No bootstrap required, opened a new log
I20260928 15:00:22.422821 15296 log.cc:824] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:22.422894 15295 ts_tablet_manager.cc:1397] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 15:00:22.425454 15295 raft_consensus.cc:348] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.425623 15295 raft_consensus.cc:374] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:22.425663 15295 raft_consensus.cc:729] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e17c9ff2fe44b9db40fd6ee5a307755, State: Initialized, Role: FOLLOWER
I20260928 15:00:22.425779 15295 consensus_queue.cc:260] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.426005 15295 ts_tablet_manager.cc:1428] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:22.426487 14994 heartbeater.cc:503] Master 127.1.252.254:37981 was elected leader, sending a full tablet report...
I20260928 15:00:22.426900 15299 tablet_bootstrap.cc:492] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Bootstrap starting.
I20260928 15:00:22.427729 15299 tablet_bootstrap.cc:654] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:22.428056 15299 log.cc:824] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:22.428750 15299 tablet_bootstrap.cc:492] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: No bootstrap required, opened a new log
I20260928 15:00:22.428830 15299 ts_tablet_manager.cc:1397] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:00:22.428844 15296 tablet_bootstrap.cc:492] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: No bootstrap required, opened a new log
I20260928 15:00:22.428907 15296 ts_tablet_manager.cc:1397] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 15:00:22.431070 15299 raft_consensus.cc:348] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.431212 15299 raft_consensus.cc:374] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:22.431286 15299 raft_consensus.cc:729] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd16cb438f7046fb83a37710f5ec57fc, State: Initialized, Role: FOLLOWER
I20260928 15:00:22.431378 15296 raft_consensus.cc:348] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.431463 15299 consensus_queue.cc:260] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.431586 15296 raft_consensus.cc:374] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:22.431640 15296 raft_consensus.cc:729] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356500cf902c48a59d97486b801499ce, State: Initialized, Role: FOLLOWER
I20260928 15:00:22.431814 15299 ts_tablet_manager.cc:1428] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:00:22.431793 15296 consensus_queue.cc:260] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.431967 15262 heartbeater.cc:503] Master 127.1.252.254:37981 was elected leader, sending a full tablet report...
I20260928 15:00:22.432087 15296 ts_tablet_manager.cc:1428] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:22.432112 15128 heartbeater.cc:503] Master 127.1.252.254:37981 was elected leader, sending a full tablet report...
W20260928 15:00:22.453094 14995 tablet.cc:2369] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:00:22 dist-test-slave-sf70 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for HTTP/127.1.252.212@KRBTEST.COM
W20260928 15:00:22.614596 15263 tablet.cc:2369] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:00:22.656409 15129 tablet.cc:2369] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:22.719844 15300 raft_consensus.cc:482] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:22.720039 15300 raft_consensus.cc:504] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.720533 15300 leader_election.cc:288] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd16cb438f7046fb83a37710f5ec57fc (127.1.252.195:42885), 356500cf902c48a59d97486b801499ce (127.1.252.194:42391)
I20260928 15:00:22.724712 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.720803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:37247 (local address 127.1.252.194:42391)
0928 15:00:22.720855 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.720857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.721194 (+ 337us) server_negotiation.cc:407] Connection header received
0928 15:00:22.721256 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.721257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.721267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.721321 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:22.721707 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.722260 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.723591 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.724205 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.724262 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.724270 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.724619 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.724659 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 15:00:22.724865 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.720716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42391 (local address 127.1.252.193:37247)
0928 15:00:22.721088 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:22.721113 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:22.721187 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:22.721378 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:22.721379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:22.721381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:22.721612 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.721614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.722341 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.722341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.723493 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.723496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.724361 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.724363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.724458 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.724463 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:00:22.724583 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:00:22.724627 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":5,"thread_start_us":255,"threads_started":1}
I20260928 15:00:22.725020 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.720847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:50947 (local address 127.1.252.195:42885)
0928 15:00:22.720890 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.720891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.721255 (+ 364us) server_negotiation.cc:407] Connection header received
0928 15:00:22.721283 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.721283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.721292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.721342 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:22.721706 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.722432 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.723822 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.724484 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.724573 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.724583 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.724913 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.724958 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:00:22.725128 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.720722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:42885 (local address 127.1.252.193:50947)
0928 15:00:22.721113 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:22.721122 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:22.721187 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:22.721384 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:22.721384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:22.721386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:22.721587 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.721590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.722539 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.722540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.723717 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:22.723720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.724615 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.724616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:22.724726 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.724738 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:00:22.724902 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 15:00:22.724945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":17,"thread_start_us":266,"threads_started":1}
I20260928 15:00:22.725319 15082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9762a770404cde831dc3e478cf3796" candidate_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356500cf902c48a59d97486b801499ce" is_pre_election: true
I20260928 15:00:22.725490 15082 raft_consensus.cc:2454] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e17c9ff2fe44b9db40fd6ee5a307755 in term 0.
I20260928 15:00:22.725736 14884 leader_election.cc:302] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356500cf902c48a59d97486b801499ce, 9e17c9ff2fe44b9db40fd6ee5a307755; no voters:
I20260928 15:00:22.725703 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9762a770404cde831dc3e478cf3796" candidate_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd16cb438f7046fb83a37710f5ec57fc" is_pre_election: true
I20260928 15:00:22.725888 15208 raft_consensus.cc:2454] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e17c9ff2fe44b9db40fd6ee5a307755 in term 0.
I20260928 15:00:22.725891 15300 raft_consensus.cc:2781] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:22.725945 15300 raft_consensus.cc:482] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:22.725977 15300 raft_consensus.cc:3037] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:22.726956 15300 raft_consensus.cc:504] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.727101 15300 leader_election.cc:288] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [CANDIDATE]: Term 1 election: Requested vote from peers bd16cb438f7046fb83a37710f5ec57fc (127.1.252.195:42885), 356500cf902c48a59d97486b801499ce (127.1.252.194:42391)
I20260928 15:00:22.727221 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9762a770404cde831dc3e478cf3796" candidate_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd16cb438f7046fb83a37710f5ec57fc"
I20260928 15:00:22.727257 15082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9762a770404cde831dc3e478cf3796" candidate_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356500cf902c48a59d97486b801499ce"
I20260928 15:00:22.727293 15208 raft_consensus.cc:3037] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:22.727353 15082 raft_consensus.cc:3037] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:22.728376 15082 raft_consensus.cc:2454] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e17c9ff2fe44b9db40fd6ee5a307755 in term 1.
I20260928 15:00:22.728437 15208 raft_consensus.cc:2454] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e17c9ff2fe44b9db40fd6ee5a307755 in term 1.
I20260928 15:00:22.728538 14884 leader_election.cc:302] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356500cf902c48a59d97486b801499ce, 9e17c9ff2fe44b9db40fd6ee5a307755; no voters:
I20260928 15:00:22.728647 15300 raft_consensus.cc:2781] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:22.728811 15300 raft_consensus.cc:686] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [term 1 LEADER]: Becoming Leader. State: Replica: 9e17c9ff2fe44b9db40fd6ee5a307755, State: Running, Role: LEADER
I20260928 15:00:22.728911 15300 consensus_queue.cc:237] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } }
I20260928 15:00:22.729733 14796 catalog_manager.cc:5495] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e17c9ff2fe44b9db40fd6ee5a307755 (127.1.252.193). New cstate: current_term: 1 leader_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e17c9ff2fe44b9db40fd6ee5a307755" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 } health_report { overall_health: UNKNOWN } } }
I20260928 15:00:22.765458 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.763076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.1.252.193:34513)
0928 15:00:22.763143 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:22.763145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:22.763153 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:00:22.763188 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:22.763189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:22.763206 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:22.763271 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:22.763442 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.763986 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.764773 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:22.764966 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:22.765024 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:22.765189 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:22.765287 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:22.765347 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:22.765381 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 15:00:22.765399 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:00:22.767781 15208 raft_consensus.cc:1261] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 1 FOLLOWER]: Refusing update from remote peer 9e17c9ff2fe44b9db40fd6ee5a307755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:22.767779 15082 raft_consensus.cc:1261] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 1 FOLLOWER]: Refusing update from remote peer 9e17c9ff2fe44b9db40fd6ee5a307755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:22.768170 15315 consensus_queue.cc:1035] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd16cb438f7046fb83a37710f5ec57fc" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:22.768319 15300 consensus_queue.cc:1035] T 0c9762a770404cde831dc3e478cf3796 P 9e17c9ff2fe44b9db40fd6ee5a307755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "356500cf902c48a59d97486b801499ce" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:22.771490 15319 mvcc.cc:205] Tried to move back new op lower bound from 7334328822853980160 to 7334328822698209280. Current Snapshot: MvccSnapshot[applied={T|T < 7334328822853980160}]
I20260928 15:00:22.771394 15318 mvcc.cc:205] Tried to move back new op lower bound from 7334328822853980160 to 7334328822698209280. Current Snapshot: MvccSnapshot[applied={T|T < 7334328822853980160}]
I20260928 15:00:22.772149 15320 mvcc.cc:205] Tried to move back new op lower bound from 7334328822853980160 to 7334328822698209280. Current Snapshot: MvccSnapshot[applied={T|T < 7334328822853980160}]
I20260928 15:00:22.774580 14796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:22.774660 14796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:22.775414 14796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9dd6c88a5ff49759f5af59fa8ce4a2c: Sending DeleteTablet for 3 replicas of tablet 0c9762a770404cde831dc3e478cf3796
I20260928 15:00:22.775868 15062 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9762a770404cde831dc3e478cf3796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:22 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:41752
I20260928 15:00:22.775873 15192 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9762a770404cde831dc3e478cf3796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:22 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:58810
I20260928 15:00:22.775907 14910 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9762a770404cde831dc3e478cf3796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:22 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:38016
I20260928 15:00:22.776319 15328 tablet_replica.cc:330] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: stopping tablet replica
I20260928 15:00:22.776340 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14834
I20260928 15:00:22.776423 15328 raft_consensus.cc:2229] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:22.776547 15328 raft_consensus.cc:2258] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:22.777403 15328 ts_tablet_manager.cc:1905] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:22.778303 15329 tablet_replica.cc:330] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: stopping tablet replica
I20260928 15:00:22.778407 15329 raft_consensus.cc:2229] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:22.778527 15329 raft_consensus.cc:2258] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:22.778900 15329 ts_tablet_manager.cc:1905] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:22.779323 15328 ts_tablet_manager.cc:1918] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:22.779413 15328 log.cc:1196] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/0c9762a770404cde831dc3e478cf3796
I20260928 15:00:22.779867 15328 ts_tablet_manager.cc:1939] T 0c9762a770404cde831dc3e478cf3796 P bd16cb438f7046fb83a37710f5ec57fc: Deleting consensus metadata
I20260928 15:00:22.780413 14782 catalog_manager.cc:4841] TS bd16cb438f7046fb83a37710f5ec57fc (127.1.252.195:42885): tablet 0c9762a770404cde831dc3e478cf3796 (table test-table [id=5e2d6fcd72c740968e8b55663bb9126f]) successfully deleted
I20260928 15:00:22.780500 15329 ts_tablet_manager.cc:1918] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:22.780558 15329 log.cc:1196] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/0c9762a770404cde831dc3e478cf3796
I20260928 15:00:22.780802 15329 ts_tablet_manager.cc:1939] T 0c9762a770404cde831dc3e478cf3796 P 356500cf902c48a59d97486b801499ce: Deleting consensus metadata
I20260928 15:00:22.781239 14783 catalog_manager.cc:4841] TS 356500cf902c48a59d97486b801499ce (127.1.252.194:42391): tablet 0c9762a770404cde831dc3e478cf3796 (table test-table [id=5e2d6fcd72c740968e8b55663bb9126f]) successfully deleted
W20260928 15:00:22.781921 14783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:34513 (error 108)
I20260928 15:00:22.782162 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14998
I20260928 15:00:22.787310 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15131
I20260928 15:00:22.787935 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:22.787377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:34513 (local address 127.0.0.1:38036)
0928 15:00:22.787826 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:22.787875 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260928 15:00:22.788717 14783 catalog_manager.cc:4584] TS 9e17c9ff2fe44b9db40fd6ee5a307755 (127.1.252.193:34513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c9762a770404cde831dc3e478cf3796: Network error: Client connection negotiation failed: client connection to 127.1.252.193:34513: connect: Connection refused (error 111)
I20260928 15:00:22.792477 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 14763
I20260928 15:00:22.799309 2035 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:00:23.143839 2035 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:00:23.143993 2035 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:00:23.488109 2035 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:00:23.488 UTC [13829] LOG: received smart shutdown request
2026-09-28 15:00:23.493 UTC [13829] LOG: background worker "logical replication launcher" (PID 13838) exited with exit code 1
2026-09-28 15:00:23.493 UTC [13833] LOG: shutting down
2026-09-28 15:00:23.669 UTC [13829] LOG: database system is shut down
2026-09-28T15:00:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48496 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:00:23 dist-test-slave-sf70 krb5kdc[15340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:00:23 dist-test-slave-sf70 krb5kdc[15340](info): set up 2 sockets
Sep 28 15:00:23 dist-test-slave-sf70 krb5kdc[15340](info): commencing operation
krb5kdc: starting...
W20260928 15:00:24.817428 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:00:24 dist-test-slave-sf70 krb5kdc[15340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:00:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:00:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:00:24.940016 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41995
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41995,127.1.252.253:42021,127.1.252.252:46047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:25.019351 15353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:25.019586 15353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:25.019624 15353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:25.020766 15353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:25.020833 15353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:25.020856 15353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:25.020877 15353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:25.020903 15353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:25.022415 15353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41995,127.1.252.253:42021,127.1.252.252:46047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:25.022680 15353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:25.022890 15353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:25.027643 15358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:25.027766 15359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:25.027824 15353 server_base.cc:1034] running on GCE node
W20260928 15:00:25.027647 15361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:25.028257 15353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:25.028553 15353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:25.029712 15353 hybrid_clock.cc:648] HybridClock initialized: now 1790607625029697 us; error 27 us; skew 500 ppm
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:25.032191 15353 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:25.032543 15353 webserver.cc:460] Webserver started at http://127.1.252.254:36159/ using document root <none> and password file <none>
I20260928 15:00:25.032773 15353 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:25.032819 15353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:25.032912 15353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:25.033985 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "4d543ce46573153f4c223fc7976872de"
server_key_iv: "4d2f63526488da8e0264f57836eccd6d"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.034368 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "4d543ce46573153f4c223fc7976872de"
server_key_iv: "4d2f63526488da8e0264f57836eccd6d"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.035768 15353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:00:25.036605 15368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:25.036842 15353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:00:25.036932 15353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "4d543ce46573153f4c223fc7976872de"
server_key_iv: "4d2f63526488da8e0264f57836eccd6d"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.036988 15353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:25.055094 15353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:25.057013 15353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:25.057122 15353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:25.061157 15353 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:41995
I20260928 15:00:25.061192 15420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:41995 every 8 connection(s)
I20260928 15:00:25.061574 15353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:25.062466 15421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:25.063987 15421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:00:25.064379 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15353
I20260928 15:00:25.064478 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:00:25.064646 2035 external_mini_cluster.cc:1428] Setting key 677e16ce4f593f15660815edbd4258f4
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.253@KRBTEST.COM; defaulting to no policy
I20260928 15:00:25.074203 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.064842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.1.252.254:41995)
0928 15:00:25.065133 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.065136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.065162 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:00:25.065995 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.065997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.066014 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.066166 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.066697 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.067487 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.068517 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.068834 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.068959 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.070975 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.070988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.070988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.070994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.072572 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.072960 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.072963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.072963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.073042 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.073269 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.073270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.073270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.073467 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.073558 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.073742 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.073946 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":263,"thread_start_us":97,"threads_started":1}
I20260928 15:00:25.074205 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.064835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41995 (local address 127.0.0.1:35872)
0928 15:00:25.065028 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.065078 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.065823 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.066172 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.066174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.066443 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.066592 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.066596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.067602 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.067603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.068415 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.068418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.068932 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.068933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.068996 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.069561 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.069569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.070879 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.072673 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.072679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.072681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.072867 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.073111 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.073112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.073112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.073203 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.073539 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.073544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.073603 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.073717 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.073986 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":128,"spinlock_wait_cycles":8064,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.253@KRBTEST.COM" created.
I20260928 15:00:25.075296 15421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:00:25.075688 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.075539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56730)
0928 15:00:25.075628 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.075668 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 15:00:25.075839 15370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
I20260928 15:00:25.076328 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.076214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56732)
0928 15:00:25.076273 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.076302 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 15:00:25.076432 15421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:42021: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab.
Entry for principal kudu/127.1.252.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab.
I20260928 15:00:25.091004 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:42021
--webserver_interface=127.1.252.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab
--principal=kudu/127.1.252.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41995,127.1.252.253:42021,127.1.252.252:46047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:25.125656 15421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } attempt: 1
I20260928 15:00:25.126149 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.126003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56736)
0928 15:00:25.126077 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.126118 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:00:25.126502 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.126406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56746)
0928 15:00:25.126453 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.126477 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 15:00:25.126610 15421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:42021: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
W20260928 15:00:25.170321 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:25.170559 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:25.170595 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:25.171756 15428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:25.171824 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:25.171847 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:25.171868 15428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:25.171921 15428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:25.173393 15428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41995,127.1.252.253:42021,127.1.252.252:46047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab
--principal=kudu/127.1.252.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.252.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:25.173663 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:25.173882 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:25.178891 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:25.178970 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:25.178916 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:25.179087 15428 server_base.cc:1034] running on GCE node
I20260928 15:00:25.179358 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:25.179594 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:25.180747 15428 hybrid_clock.cc:648] HybridClock initialized: now 1790607625180736 us; error 28 us; skew 500 ppm
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:25.183127 15428 init.cc:377] Logged in from keytab as kudu/127.1.252.253@KRBTEST.COM (short username kudu)
I20260928 15:00:25.183564 15428 webserver.cc:460] Webserver started at http://127.1.252.253:37735/ using document root <none> and password file <none>
I20260928 15:00:25.183786 15428 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:25.183830 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:25.184007 15428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:25.185109 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data/instance:
uuid: "51ac5f294f9b4ed4b565fe6dd31f4880"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "fd7235392492068e728ce9ee2be3d33e"
server_key_iv: "79bfe0d605b68fa677f413e9e5e47e5a"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.185464 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal/instance:
uuid: "51ac5f294f9b4ed4b565fe6dd31f4880"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "fd7235392492068e728ce9ee2be3d33e"
server_key_iv: "79bfe0d605b68fa677f413e9e5e47e5a"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.186812 15428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:00:25.187592 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:25.187801 15428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:00:25.187937 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal
uuid: "51ac5f294f9b4ed4b565fe6dd31f4880"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "fd7235392492068e728ce9ee2be3d33e"
server_key_iv: "79bfe0d605b68fa677f413e9e5e47e5a"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.188033 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:25.194748 15421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } attempt: 2
I20260928 15:00:25.195152 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.195008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56758)
0928 15:00:25.195065 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.195106 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 15:00:25.195497 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.195396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56772)
0928 15:00:25.195442 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.195466 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 15:00:25.195577 15421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:42021: Network error: Client connection negotiation failed: client connection to 127.1.252.253:42021: connect: Connection refused (error 111)
I20260928 15:00:25.199597 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:25.201567 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:25.201731 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:25.206113 15428 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.253:42021
I20260928 15:00:25.206141 15495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.253:42021 every 8 connection(s)
I20260928 15:00:25.206485 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
I20260928 15:00:25.207305 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:25.208861 15496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:25.215567 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15428
I20260928 15:00:25.215689 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-1/wal/instance
I20260928 15:00:25.215899 2035 external_mini_cluster.cc:1428] Setting key d7581f130eb82ca458a6c3c401c9f914
I20260928 15:00:25.217413 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.209523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.1.252.254:41995)
0928 15:00:25.209592 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.209594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.209790 (+ 196us) server_negotiation.cc:407] Connection header received
0928 15:00:25.210215 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.210216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.210227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.210291 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.210848 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.211334 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.212349 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.212545 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.212603 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.214514 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.214515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.214516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.214522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.216115 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.216474 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.216475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.216476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.216541 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.216793 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.216794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.216794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.216938 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.217006 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.217243 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.217330 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 15:00:25.217581 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.209495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41995 (local address 127.0.0.1:35874)
0928 15:00:25.209636 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.209717 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.210133 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.210340 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.210342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.210561 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.210769 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.210773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.211411 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.211412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.212277 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.212279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.212615 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.212615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.212738 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.213141 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.213148 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.214410 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.216194 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.216201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.216202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.216399 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.216607 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.216607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.216607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.216725 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.217002 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.217006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.217107 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.217215 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.217357 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260928 15:00:25.218195 15496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.252@KRBTEST.COM; defaulting to no policy
I20260928 15:00:25.225306 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.218445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56786)
0928 15:00:25.218491 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.218500 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.218534 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.218827 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.218828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.218966 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.219008 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.219009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.219867 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.219868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.220386 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.220387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.220862 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.220863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.220928 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.221330 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.221335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.222422 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.224293 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.224294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.224294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.224451 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.224671 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.224672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.224672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.224723 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.224997 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.224998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.225034 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.225104 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.225195 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 15:00:25.225381 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.218526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.1.252.253:42021)
0928 15:00:25.218700 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.218704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.218718 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:00:25.218734 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.218736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.218746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.218826 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.219085 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.219764 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.220483 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.220772 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.220861 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.222503 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.222511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.222511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.222517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.224197 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.224541 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.224544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.224544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.224598 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.224794 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.224795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.224796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.224925 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.225016 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.225133 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.225245 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260928 15:00:25.226104 15496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.252@KRBTEST.COM" created.
I20260928 15:00:25.226461 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.226351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:50978)
0928 15:00:25.226383 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.226429 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 15:00:25.226642 15446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
I20260928 15:00:25.227118 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.227002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:50994)
0928 15:00:25.227067 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.227095 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:00:25.227218 15496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:46047: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab.
Entry for principal kudu/127.1.252.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab.
I20260928 15:00:25.242965 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:46047
--webserver_interface=127.1.252.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab
--principal=kudu/127.1.252.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:41995,127.1.252.253:42021,127.1.252.252:46047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:25.276415 15496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } attempt: 1
I20260928 15:00:25.276921 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.276760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51008)
0928 15:00:25.276830 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.276880 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:00:25.277330 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.277205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51018)
0928 15:00:25.277259 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.277296 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 15:00:25.277460 15496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:46047: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
I20260928 15:00:25.286717 15421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } attempt: 3
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:00:25.299983 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.287080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.1.252.253:42021)
0928 15:00:25.287144 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.287146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.287157 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:00:25.287258 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.287258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.287271 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.287343 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.287674 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.288250 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.288934 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.289148 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.289229 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.297441 (+ 8212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.297443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.297443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.297450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.298857 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.299244 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.299245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.299245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.299295 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.299487 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.299488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.299488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.299586 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.299656 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.299812 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.299875 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:00:25.300007 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.287025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56796)
0928 15:00:25.287075 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.287090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.287167 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.287348 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.287348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.287509 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.287574 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.287575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.288336 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.288337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.288846 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.288848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.289225 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.289227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.289307 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.295831 (+ 6524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.295836 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.297354 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.298945 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.298947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.298947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.299169 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.299359 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.299360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.299360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.299429 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.299652 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.299653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.299713 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.299788 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.299909 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 15:00:25.300388 15421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:00:25.300768 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.300638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51022)
0928 15:00:25.300699 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.300740 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 15:00:25.301111 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.301015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51034)
0928 15:00:25.301065 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.301092 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:00:25.301206 15421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:46047: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
W20260928 15:00:25.323794 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:25.324060 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:25.324084 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:25.325258 15503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:25.325299 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:25.325313 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:25.325326 15503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:25.325374 15503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:25.326831 15503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:41995,127.1.252.253:42021,127.1.252.252:46047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab
--principal=kudu/127.1.252.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:46047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.252.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:25.327068 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:25.327319 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:25.332058 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:25.332064 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:25.332219 15503 server_base.cc:1034] running on GCE node
W20260928 15:00:25.332093 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:25.332343 15421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } attempt: 1
I20260928 15:00:25.332561 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:25.332795 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.332657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51050)
0928 15:00:25.332726 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.332762 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 15:00:25.332870 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:25.333117 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.333038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51062)
0928 15:00:25.333069 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.333093 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 15:00:25.333222 15421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:46047: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
I20260928 15:00:25.334023 15503 hybrid_clock.cc:648] HybridClock initialized: now 1790607625333999 us; error 34 us; skew 500 ppm
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:25.339311 15503 init.cc:377] Logged in from keytab as kudu/127.1.252.252@KRBTEST.COM (short username kudu)
I20260928 15:00:25.339859 15503 webserver.cc:460] Webserver started at http://127.1.252.252:41563/ using document root <none> and password file <none>
I20260928 15:00:25.340133 15503 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:25.340200 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:25.340346 15503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:25.341800 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data/instance:
uuid: "3ff5ce1d0b254988b9e8430e29ab0012"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "9c25745b8a7beaa91bd8d9099c5cabe5"
server_key_iv: "8f21878290ef66bd2a4c451db0bcec79"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.342180 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal/instance:
uuid: "3ff5ce1d0b254988b9e8430e29ab0012"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "9c25745b8a7beaa91bd8d9099c5cabe5"
server_key_iv: "8f21878290ef66bd2a4c451db0bcec79"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.343966 15503 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 15:00:25.345109 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:25.345307 15503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:25.345410 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal
uuid: "3ff5ce1d0b254988b9e8430e29ab0012"
format_stamp: "Formatted at 2026-09-28 15:00:25 on dist-test-slave-sf70"
server_key: "9c25745b8a7beaa91bd8d9099c5cabe5"
server_key_iv: "8f21878290ef66bd2a4c451db0bcec79"
server_key_version: "encryptionkey@0"
I20260928 15:00:25.345494 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:25.345615 15496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } attempt: 2
I20260928 15:00:25.346004 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.345879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51076)
0928 15:00:25.345929 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.345970 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 15:00:25.346549 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.346311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51086)
0928 15:00:25.346493 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.346522 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260928 15:00:25.346720 15496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:46047: Network error: Client connection negotiation failed: client connection to 127.1.252.252:46047: connect: Connection refused (error 111)
I20260928 15:00:25.385650 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:25.387485 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:25.387593 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:25.391721 15503 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.252:46047
I20260928 15:00:25.391774 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.252:46047 every 8 connection(s)
I20260928 15:00:25.392119 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
I20260928 15:00:25.392976 15571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:25.394623 15571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:00:25.397687 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15503
I20260928 15:00:25.397807 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607474232031-2035-0/minicluster-data/master-2/wal/instance
I20260928 15:00:25.398015 2035 external_mini_cluster.cc:1428] Setting key b60f5e71a051c08331f2f323b67681cf
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:00:25.404109 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.395373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.1.252.254:41995)
0928 15:00:25.395429 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.395431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.395580 (+ 149us) server_negotiation.cc:407] Connection header received
0928 15:00:25.396205 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.396206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.396217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.396327 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.396978 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.397461 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.398779 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.398963 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.399016 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.401231 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.401232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.401232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.401238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.402670 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.403094 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.403095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.403095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.403153 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.403445 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.403446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.403447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.403561 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.403640 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.403937 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.404022 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:00:25.404282 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.395294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:41995 (local address 127.0.0.1:35890)
0928 15:00:25.395458 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.395499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.396059 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.396425 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.396429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.396692 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.396881 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.396885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.397575 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.397577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.398700 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.398702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.399033 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.399033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.399141 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.399633 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.399641 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.401118 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.402774 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.402782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.402784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.403001 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.403230 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.403231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.403232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.403338 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.403632 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.403638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.403778 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.403881 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.404043 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260928 15:00:25.404821 15571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:00:25.407027 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.398405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.1.252.252:46047)
0928 15:00:25.398551 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.398555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.398602 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:00:25.398670 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.398673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.398697 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.398767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.399613 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.400418 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.401169 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.401477 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.401527 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.403774 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.403782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.403782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.403789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.405825 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.406215 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.406217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.406217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.406266 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.406471 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.406472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.406472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.406630 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.406815 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.406856 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.406928 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 15:00:25.408349 15421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } attempt: 2
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:00:25.412310 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.405069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:42021 (local address 127.0.0.1:56810)
0928 15:00:25.405200 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.405212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.405264 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.405385 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.405386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.405509 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.405543 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.405544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.406363 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.406364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.407097 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.407099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.407484 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.407485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.407550 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.407997 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.408003 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.409480 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.411140 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.411141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.411141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.411377 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.411572 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.411573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.411573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.411633 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.411997 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.411997 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.412033 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.412131 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.412223 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 15:00:25.412326 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.405143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.1.252.253:42021)
0928 15:00:25.405194 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.405196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.405249 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:00:25.405324 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.405324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.405333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.405372 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.405643 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.406266 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.407177 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.407406 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.407486 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.409577 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.409578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.409578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.409585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.411076 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.411451 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.411452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.411452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.411501 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.411737 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.411738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.411739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.411914 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.412000 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.412166 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.412240 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 15:00:25.413034 15571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:25.417454 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.408592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51104)
0928 15:00:25.408632 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.408642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.408681 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.408798 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.408799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.408950 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.408982 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.408984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.410039 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.410040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.410848 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.410851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.411327 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.411328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.411387 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.412049 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.412055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.413519 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.416135 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.416136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.416137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.416375 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.416579 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.416579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.416580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.416659 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.417161 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.417162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.417202 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.417270 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.417354 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 15:00:25.417757 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.408639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.1.252.252:46047)
0928 15:00:25.408685 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.408687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.408719 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:00:25.408737 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.408738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.408747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.408794 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.409057 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.409713 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.410934 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.411255 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.411301 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.413605 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.413606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.413606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.413614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.416058 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.416440 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.416441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.416442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.416503 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.416734 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.416735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.416735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.416839 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.416924 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.417602 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.417675 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 15:00:25.417922 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.407973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.1.252.254:41995)
0928 15:00:25.408011 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.408013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.408020 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:00:25.408039 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.408039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.408048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.408100 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.408866 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.409474 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.410393 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.410690 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.410761 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.414388 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.414389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.414390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.414396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.416261 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.416620 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.416621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.416622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.416698 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.416904 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.416905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.416905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.417010 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.417696 (+ 686us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.417741 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.417819 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:00:25.419804 15421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: Bootstrap starting.
I20260928 15:00:25.420550 15421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:25.420889 15421 log.cc:824] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:25.422087 15421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: No bootstrap required, opened a new log
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:00:25.423261 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.413307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51110)
0928 15:00:25.413364 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.413375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.413418 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.413739 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.413739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.413881 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.413911 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.413913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.414717 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.414718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.415462 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.415467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.416699 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.416700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.416768 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.417162 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.417167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.418790 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.422339 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.422340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.422340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.422504 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.422703 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.422703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.422704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.422756 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.423020 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.423021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.423058 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.423121 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.423207 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:00:25.423552 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.413369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.1.252.252:46047)
0928 15:00:25.413626 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.413628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.413633 (+ 5us) server_negotiation.cc:407] Connection header received
0928 15:00:25.413648 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.413648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.413655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.413737 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.413997 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.414628 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.415574 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.416625 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.416693 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.418868 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.418869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.418869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.418875 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.422254 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.422586 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.422587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.422588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.422641 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.422835 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.422836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.422836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.422954 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.423025 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.423383 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.423451 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":80,"threads_started":1}
I20260928 15:00:25.424687 15421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.424821 15421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:25.424865 15421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d8bdb7a87f4050a5cbe3dbb7c2c708, State: Initialized, Role: FOLLOWER
I20260928 15:00:25.424988 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: Bootstrap starting.
I20260928 15:00:25.424983 15421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.425263 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.425329 15421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:25.425391 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:00:25.425714 15571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:25.426048 15571 log.cc:824] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:25.426692 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.418471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.1.252.253:42021)
0928 15:00:25.418518 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.418520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.418541 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:00:25.418565 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.418566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.418576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.418632 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.419223 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.419644 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.420518 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.420727 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.420780 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.423610 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.423612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.423612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.423619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.425256 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.425664 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.425665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.425665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.425725 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.425944 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.425945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.425945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.426096 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.426288 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.426526 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.426608 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:00:25.426800 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: No bootstrap required, opened a new log
I20260928 15:00:25.428128 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 15:00:25.428597 15590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:00:25.428656 15590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:00:25.428973 15571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.429106 15571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:25.429147 15571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff5ce1d0b254988b9e8430e29ab0012, State: Initialized, Role: FOLLOWER
I20260928 15:00:25.429253 15571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.429535 15571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:25.429896 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.430061 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:00:25.431299 15608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:00:25.431375 15608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:00:25.429634 15353 master_runner.cc:186] Error getting master registration for 127.1.252.253:42021: OK, instance_id { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" instance_seqno: 1790607625204943 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:00:25.432014 15353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:00:25.433507 15503 master_runner.cc:186] Error getting master registration for 127.1.252.253:42021: OK, instance_id { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" instance_seqno: 1790607625204943 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:00:25.435676 15503 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:00:25.437875 15496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } attempt: 3
I20260928 15:00:25.439529 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.429587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.1.252.254:41995)
0928 15:00:25.429619 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.429620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.431978 (+ 2358us) server_negotiation.cc:407] Connection header received
0928 15:00:25.432004 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.432005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.432016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.432079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.432787 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.433525 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.434559 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.434835 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.434889 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.436398 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.436399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.436399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.436405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.437953 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.438477 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.438478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.438479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.438537 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.438740 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.438741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.438741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.438870 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.438954 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.439358 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.439434 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 15:00:25.441829 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.432820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.1.252.252:46047)
0928 15:00:25.432853 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.432855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.433373 (+ 518us) server_negotiation.cc:407] Connection header received
0928 15:00:25.433415 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.433416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.433426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.433477 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.434166 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.434865 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.435984 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.436320 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.436359 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.438876 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.438877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.438878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.438884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.440589 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.440975 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.440976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.440977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.441031 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.441232 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.441232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.441233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.441346 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.441446 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.441666 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.441748 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 15:00:25 dist-test-slave-sf70 krb5kdc[15340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:00:25.449437 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.438081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:46047 (local address 127.0.0.1:51126)
0928 15:00:25.438121 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:25.438132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:25.438172 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:25.438330 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:25.438331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:25.438488 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:25.438529 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.438531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.439871 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.439872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.440636 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:25.440638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.441115 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.441116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:25.441177 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.441559 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:25.441564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:25.443024 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:25.448274 (+ 5250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.448275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.448275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.448607 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.448825 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:25.448826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:25.448826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:25.448881 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:25.449152 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:25.449153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:25.449193 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:00:25.449253 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 15:00:25.449340 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 15:00:25.449744 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.438167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.1.252.252:46047)
0928 15:00:25.438202 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.438204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.438212 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:00:25.438230 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.438231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.438239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.438291 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.438607 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.439583 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.440736 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.441022 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.441070 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.443113 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.443114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.443114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.443121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.445018 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.448697 (+ 3679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.448699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.448699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.448757 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.448943 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.448943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.448944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.449056 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.449118 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.449587 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.449661 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 15:00:25.450407 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:25.433152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.1.252.253:42021)
0928 15:00:25.433202 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:25.433204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:25.438858 (+ 5654us) server_negotiation.cc:407] Connection header received
0928 15:00:25.438943 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:25.438943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:25.438952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:25.438978 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:25.439565 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.440355 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.443991 (+ 3636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:25.444277 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:25.444319 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:25.446800 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:25.446801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:25.446801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:25.446809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:25.448807 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.449200 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.449201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.449202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.449255 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:25.449449 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:25.449450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:25.449450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:25.449578 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:25.449646 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:25.450244 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 15:00:25.450319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 15:00:25.451253 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880: Bootstrap starting.
I20260928 15:00:25.451977 15496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:25.452284 15496 log.cc:824] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:25.453123 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880: No bootstrap required, opened a new log
I20260928 15:00:25.455090 15496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.455238 15496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:25.455273 15496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ac5f294f9b4ed4b565fe6dd31f4880, State: Initialized, Role: FOLLOWER
I20260928 15:00:25.455389 15496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.455689 15496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:25.459921 15428 master_runner.cc:263] No leader master found from master 51ac5f294f9b4ed4b565fe6dd31f4880
I20260928 15:00:25.459968 15428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:00:25.464076 15613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.464176 15613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:00:25.464854 15624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:00:25.464908 15624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:00:25.548374 15613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:25.548514 15613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.548969 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" is_pre_election: true
I20260928 15:00:25.549153 15395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ac5f294f9b4ed4b565fe6dd31f4880 in term 0.
I20260928 15:00:25.549350 15447 leader_election.cc:302] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ac5f294f9b4ed4b565fe6dd31f4880, 54d8bdb7a87f4050a5cbe3dbb7c2c708; no voters:
I20260928 15:00:25.549813 15545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" is_pre_election: true
I20260928 15:00:25.549957 15545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ac5f294f9b4ed4b565fe6dd31f4880 in term 0.
I20260928 15:00:25.550174 15613 leader_election.cc:288] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54d8bdb7a87f4050a5cbe3dbb7c2c708 (127.1.252.254:41995), 3ff5ce1d0b254988b9e8430e29ab0012 (127.1.252.252:46047)
I20260928 15:00:25.550276 15613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:25.550326 15613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:25.550352 15613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:25.551290 15613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.551541 15395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708"
I20260928 15:00:25.551620 15395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:25.552994 15395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ac5f294f9b4ed4b565fe6dd31f4880 in term 1.
I20260928 15:00:25.553174 15447 leader_election.cc:302] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ac5f294f9b4ed4b565fe6dd31f4880, 54d8bdb7a87f4050a5cbe3dbb7c2c708; no voters:
I20260928 15:00:25.553381 15545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff5ce1d0b254988b9e8430e29ab0012"
I20260928 15:00:25.553467 15545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:25.554361 15545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ac5f294f9b4ed4b565fe6dd31f4880 in term 1.
I20260928 15:00:25.554548 15613 leader_election.cc:288] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [CANDIDATE]: Term 1 election: Requested vote from peers 54d8bdb7a87f4050a5cbe3dbb7c2c708 (127.1.252.254:41995), 3ff5ce1d0b254988b9e8430e29ab0012 (127.1.252.252:46047)
I20260928 15:00:25.554621 15613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:25.554795 15613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [term 1 LEADER]: Becoming Leader. State: Replica: 51ac5f294f9b4ed4b565fe6dd31f4880, State: Running, Role: LEADER
I20260928 15:00:25.554891 15613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } }
I20260928 15:00:25.555342 15613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ac5f294f9b4ed4b565fe6dd31f4880. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.555447 15613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:25.555706 15628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:25.555917 15628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 15:00:25.557794 15374 tablet.cc:2369] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:25.557963 15545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [term 1 FOLLOWER]: Refusing update from remote peer 51ac5f294f9b4ed4b565fe6dd31f4880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:25.558274 15626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:25.560629 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ac5f294f9b4ed4b565fe6dd31f4880. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.560719 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:00:25.561024 15395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [term 1 FOLLOWER]: Refusing update from remote peer 51ac5f294f9b4ed4b565fe6dd31f4880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:25.561271 15626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:25.562273 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ac5f294f9b4ed4b565fe6dd31f4880. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.562358 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:00:25.563746 15626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.563832 15626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:25.564291 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.564374 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:00:25.564551 15629 mvcc.cc:205] Tried to move back new op lower bound from 7334328834282897408 to 7334328834273521664. Current Snapshot: MvccSnapshot[applied={T|T < 7334328834282897408}]
I20260928 15:00:25.564647 15632 mvcc.cc:205] Tried to move back new op lower bound from 7334328834282897408 to 7334328834273521664. Current Snapshot: MvccSnapshot[applied={T|T < 7334328834282897408}]
I20260928 15:00:25.565186 15628 catalog_manager.cc:1347] Generated new cluster ID: bff5faba089f494690ccd65be3973eff
I20260928 15:00:25.565235 15628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:25.565871 15613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.565948 15613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:25.571943 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d8bdb7a87f4050a5cbe3dbb7c2c708" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 41995 } } peers { permanent_uuid: "51ac5f294f9b4ed4b565fe6dd31f4880" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 42021 } } peers { permanent_uuid: "3ff5ce1d0b254988b9e8430e29ab0012" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 46047 } } }
I20260928 15:00:25.572033 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:00:25.572104 15630 mvcc.cc:205] Tried to move back new op lower bound from 7334328834282897408 to 7334328834273521664. Current Snapshot: MvccSnapshot[applied={T|T < 7334328834282897408}]
I20260928 15:00:25.602807 15628 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:00:25.603380 15628 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:25.606905 15628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51ac5f294f9b4ed4b565fe6dd31f4880: Generated new TSK 0
I20260928 15:00:25.607085 15628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:00:26.429075 15590 catalog_manager.cc:1259] Loaded cluster ID: bff5faba089f494690ccd65be3973eff
I20260928 15:00:26.429143 15590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: loading cluster ID for follower catalog manager: success
I20260928 15:00:26.430179 15590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: acquiring CA information for follower catalog manager: success
I20260928 15:00:26.430491 15590 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54d8bdb7a87f4050a5cbe3dbb7c2c708: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:00:26.431746 15608 catalog_manager.cc:1259] Loaded cluster ID: bff5faba089f494690ccd65be3973eff
I20260928 15:00:26.431792 15608 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: loading cluster ID for follower catalog manager: success
I20260928 15:00:26.432538 15608 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: acquiring CA information for follower catalog manager: success
I20260928 15:00:26.432812 15608 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ff5ce1d0b254988b9e8430e29ab0012: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:00:26.654508 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15353
I20260928 15:00:26.660171 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15428
I20260928 15:00:26.665014 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15503
2026-09-28T15:00:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2881 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:00:26 dist-test-slave-sf70 krb5kdc[15644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:00:26 dist-test-slave-sf70 krb5kdc[15644](info): set up 2 sockets
Sep 28 15:00:26 dist-test-slave-sf70 krb5kdc[15644](info): commencing operation
krb5kdc: starting...
W20260928 15:00:27.701568 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:00:27 dist-test-slave-sf70 krb5kdc[15644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:00:27.820261 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46389
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:27.901324 15657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:27.901564 15657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:27.901603 15657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:27.902824 15657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:27.902894 15657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:27.902940 15657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:27.902983 15657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:27.903023 15657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:27.904524 15657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:46389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:27.904803 15657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:27.905035 15657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:00:27.909919 15657 server_base.cc:1034] running on GCE node
W20260928 15:00:27.909927 15662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:27.909950 15663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:27.910068 15665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:27.910315 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:27.910552 15657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:27.911734 15657 hybrid_clock.cc:648] HybridClock initialized: now 1790607627911714 us; error 30 us; skew 500 ppm
Sep 28 15:00:27 dist-test-slave-sf70 krb5kdc[15644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:27.914207 15657 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:27.914678 15657 webserver.cc:460] Webserver started at http://127.1.252.254:37883/ using document root <none> and password file <none>
I20260928 15:00:27.914923 15657 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:27.914992 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:27.915138 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:27.916270 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "947604da2bb1463cae44bddc97274d5f"
format_stamp: "Formatted at 2026-09-28 15:00:27 on dist-test-slave-sf70"
server_key: "156de0bf7c5649b307d28f4ea717e058"
server_key_iv: "fd97c79f65b6579d98a03077917dc7cc"
server_key_version: "encryptionkey@0"
I20260928 15:00:27.916625 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "947604da2bb1463cae44bddc97274d5f"
format_stamp: "Formatted at 2026-09-28 15:00:27 on dist-test-slave-sf70"
server_key: "156de0bf7c5649b307d28f4ea717e058"
server_key_iv: "fd97c79f65b6579d98a03077917dc7cc"
server_key_version: "encryptionkey@0"
I20260928 15:00:27.918004 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:27.918830 15672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:27.919131 15657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:27.919242 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "947604da2bb1463cae44bddc97274d5f"
format_stamp: "Formatted at 2026-09-28 15:00:27 on dist-test-slave-sf70"
server_key: "156de0bf7c5649b307d28f4ea717e058"
server_key_iv: "fd97c79f65b6579d98a03077917dc7cc"
server_key_version: "encryptionkey@0"
I20260928 15:00:27.919328 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:27.945782 15657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:27.948093 15657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:27.948264 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:27.952360 15657 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:46389
I20260928 15:00:27.952399 15724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:46389 every 8 connection(s)
I20260928 15:00:27.952782 15657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:27.953652 15725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:27.954866 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15657
I20260928 15:00:27.955006 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:00:27.955211 2035 external_mini_cluster.cc:1428] Setting key 3f47ca95567c63992df8a5648d3dca72
I20260928 15:00:27.956387 15725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f: Bootstrap starting.
I20260928 15:00:27.957063 15725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:27.957417 15725 log.cc:824] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:27.958169 15725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f: No bootstrap required, opened a new log
I20260928 15:00:27.960503 15725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947604da2bb1463cae44bddc97274d5f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46389 } }
I20260928 15:00:27.960687 15725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:27.960731 15725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947604da2bb1463cae44bddc97274d5f, State: Initialized, Role: FOLLOWER
I20260928 15:00:27.960871 15725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947604da2bb1463cae44bddc97274d5f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46389 } }
I20260928 15:00:27.960969 15725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:27.961012 15725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:27.961050 15725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:27.961835 15725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947604da2bb1463cae44bddc97274d5f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46389 } }
I20260928 15:00:27.961961 15725 leader_election.cc:302] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947604da2bb1463cae44bddc97274d5f; no voters:
I20260928 15:00:27.962139 15725 leader_election.cc:288] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:00:27.962252 15730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:27.962401 15730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [term 1 LEADER]: Becoming Leader. State: Replica: 947604da2bb1463cae44bddc97274d5f, State: Running, Role: LEADER
I20260928 15:00:27.962466 15725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:27.962481 15730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947604da2bb1463cae44bddc97274d5f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46389 } }
I20260928 15:00:27.962788 15730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947604da2bb1463cae44bddc97274d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947604da2bb1463cae44bddc97274d5f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46389 } } }
I20260928 15:00:27.962865 15730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [sys.catalog]: This master's current role is: LEADER
Sep 28 15:00:27 dist-test-slave-sf70 krb5kdc[15644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:27.963761 15731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947604da2bb1463cae44bddc97274d5f. Latest consensus state: current_term: 1 leader_uuid: "947604da2bb1463cae44bddc97274d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947604da2bb1463cae44bddc97274d5f" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46389 } } }
I20260928 15:00:27.964090 15736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:27.964299 15731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:27.964952 15736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:27.966455 15736 catalog_manager.cc:1347] Generated new cluster ID: 4d431e2e9ff24a08b67e6f0c35c1d55a
I20260928 15:00:27.966517 15736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:27.967423 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:27.955791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.1.252.254:46389)
0928 15:00:27.955991 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:27.955999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:27.956101 (+ 102us) server_negotiation.cc:407] Connection header received
0928 15:00:27.956643 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:27.956650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:27.956698 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:27.956823 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:27.957679 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:27.958743 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:27.959839 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:27.960390 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:27.960490 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:27.963575 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:27.963579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:27.963580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:27.963592 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:27.965803 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:27.966249 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:27.966252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:27.966252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:27.966352 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:27.966610 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:27.966611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:27.966611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:27.966849 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:27.967029 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:27.967120 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:00:27.967261 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260928 15:00:27.968465 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:00:27.976301 15736 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:00:27.976979 15736 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:27.989118 15736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 947604da2bb1463cae44bddc97274d5f: Generated new TSK 0
I20260928 15:00:27.989300 15736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:00:28.022563 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15657
2026-09-28T15:00:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1355 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:00:28 dist-test-slave-sf70 krb5kdc[15754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:00:28 dist-test-slave-sf70 krb5kdc[15754](info): set up 2 sockets
Sep 28 15:00:28 dist-test-slave-sf70 krb5kdc[15754](info): commencing operation
krb5kdc: starting...
W20260928 15:00:29.057572 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:00:29.179590 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45215
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:29.269026 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:29.269315 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:29.269361 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:29.270680 15767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:29.270761 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:29.270787 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:29.270812 15767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:29.270844 15767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:29.272531 15767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:29.272845 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:29.273104 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:29.278309 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.278379 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.278309 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:29.278338 15767 server_base.cc:1034] running on GCE node
I20260928 15:00:29.278802 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:29.279153 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:29.280321 15767 hybrid_clock.cc:648] HybridClock initialized: now 1790607629280312 us; error 36 us; skew 500 ppm
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:29.282826 15767 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:29.283270 15767 webserver.cc:460] Webserver started at http://127.1.252.254:42493/ using document root <none> and password file <none>
I20260928 15:00:29.283501 15767 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:29.283571 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:29.283730 15767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:29.284946 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "f2b67b8bca224f34a7d27e6534eb2492"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "43487e1544fda9e435fbd4b9113b55af"
server_key_iv: "d96e62a6748672b3ee528392471819d6"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.285323 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "f2b67b8bca224f34a7d27e6534eb2492"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "43487e1544fda9e435fbd4b9113b55af"
server_key_iv: "d96e62a6748672b3ee528392471819d6"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.286725 15767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:00:29.287503 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.287734 15767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:29.287846 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "f2b67b8bca224f34a7d27e6534eb2492"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "43487e1544fda9e435fbd4b9113b55af"
server_key_iv: "d96e62a6748672b3ee528392471819d6"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.287981 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:29.319049 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:29.321421 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:29.321591 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:29.325711 15767 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:45215
I20260928 15:00:29.325762 15834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:45215 every 8 connection(s)
I20260928 15:00:29.326115 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:29.327091 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:29.329497 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap starting.
I20260928 15:00:29.330073 15835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:29.330360 15835 log.cc:824] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:29.331066 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: No bootstrap required, opened a new log
I20260928 15:00:29.332739 15835 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:29.332871 15835 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:29.332893 15835 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b67b8bca224f34a7d27e6534eb2492, State: Initialized, Role: FOLLOWER
I20260928 15:00:29.333065 15835 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:29.333128 15835 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:29.333168 15835 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:29.333215 15835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:29.333994 15835 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:29.334101 15835 leader_election.cc:302] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b67b8bca224f34a7d27e6534eb2492; no voters:
I20260928 15:00:29.334280 15835 leader_election.cc:288] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:00:29.334337 15838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:29.334503 15838 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 LEADER]: Becoming Leader. State: Replica: f2b67b8bca224f34a7d27e6534eb2492, State: Running, Role: LEADER
I20260928 15:00:29.334524 15835 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:29.334591 15838 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:29.334978 15840 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b67b8bca224f34a7d27e6534eb2492. Latest consensus state: current_term: 1 leader_uuid: "f2b67b8bca224f34a7d27e6534eb2492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } } }
I20260928 15:00:29.335067 15840 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:29.335150 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15767
I20260928 15:00:29.335275 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:00:29.335284 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b67b8bca224f34a7d27e6534eb2492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } } }
I20260928 15:00:29.335354 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:29.335486 15847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:29.335502 2035 external_mini_cluster.cc:1428] Setting key 6962543f6ed783ce1fd1fe933b117f85
I20260928 15:00:29.335988 15847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:29.337826 15847 catalog_manager.cc:1347] Generated new cluster ID: 6e9b0a08eab94df598dff6046d9ead59
I20260928 15:00:29.337878 15847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:29.345408 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.335985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.1.252.254:45215)
0928 15:00:29.336275 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.336285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.336321 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:00:29.336466 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.336473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.336536 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.336703 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:29.337441 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.338486 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.339257 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.339695 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.339813 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.342177 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:29.342180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:29.342180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:29.342190 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:29.343861 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.344265 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.344268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.344268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.344365 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.344572 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.344573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.344574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.344811 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:29.344980 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.345081 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.345218 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":65,"threads_started":1}
I20260928 15:00:29.348013 15847 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:00:29.348575 15847 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:29.354990 15847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Generated new TSK 0
I20260928 15:00:29.355182 15847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 15:00:29.387094 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45215
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:29.491173 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:29.491402 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:29.491442 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:29.492758 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:29.492830 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:29.492877 15861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:00:29.494357 15861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:29.494627 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:29.494860 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:29.500665 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.500675 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.500679 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:29.501026 15861 server_base.cc:1034] running on GCE node
I20260928 15:00:29.501232 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:29.501462 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:29.502632 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790607629502617 us; error 30 us; skew 500 ppm
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:29.505224 15861 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:00:29.505692 15861 webserver.cc:460] Webserver started at http://127.1.252.193:41601/ using document root <none> and password file <none>
I20260928 15:00:29.505874 15861 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:29.505915 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:29.506000 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:29.506960 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "227bc748d83181e949d7c659105e9b66"
server_key_iv: "211748bb5b9f07ab3db084f69618ed59"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.507256 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "227bc748d83181e949d7c659105e9b66"
server_key_iv: "211748bb5b9f07ab3db084f69618ed59"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.508664 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:00:29.509443 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.509670 15861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:29.509756 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "227bc748d83181e949d7c659105e9b66"
server_key_iv: "211748bb5b9f07ab3db084f69618ed59"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.509838 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:29.529265 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:29.531390 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:29.531507 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:29.531710 15861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:29.532106 15861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:29.532145 15861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.532167 15861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:29.532214 15861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.538203 15861 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:33505
I20260928 15:00:29.538287 15990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:33505 every 8 connection(s)
I20260928 15:00:29.538610 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:29.542516 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15861
I20260928 15:00:29.542618 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:00:29.542788 2035 external_mini_cluster.cc:1428] Setting key 0851ed62f21babc363fdec733a74b14c
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:29.548782 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.539353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:41841 (local address 127.1.252.254:45215)
0928 15:00:29.539422 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.539424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.539612 (+ 188us) server_negotiation.cc:407] Connection header received
0928 15:00:29.540188 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.540189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.540203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.540265 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:29.541204 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.541707 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.542999 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.543264 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.543319 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.545780 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:29.545782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:29.545782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:29.545792 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:29.547223 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.547720 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.547721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.547722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.547787 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.548100 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.548101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.548101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.548234 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:29.548314 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.548601 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.548691 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:00:29.548966 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.539318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:41841)
0928 15:00:29.539478 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.539520 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.540076 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.540331 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.540334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.540912 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:29.541102 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.541106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.541798 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.541799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.542898 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.542901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.543341 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.543342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.543469 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.544019 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:29.544029 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:29.545675 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:29.547407 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:29.547413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:29.547415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:29.547637 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:29.547857 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:29.547858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:29.547858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:29.548011 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:29.548305 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:29.548312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:29.548460 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.548566 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.548730 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260928 15:00:29.549443 15991 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45215
I20260928 15:00:29.549553 15991 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:29.549757 15991 heartbeater.cc:511] Master 127.1.252.254:45215 requested a full tablet report, sending...
I20260928 15:00:29.550295 15799 ts_manager.cc:194] Registered new tserver with Master: e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505)
I20260928 15:00:29.550937 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:41841
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 15:00:29.568920 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45215
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:29.648432 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:29.648638 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:29.648661 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:29.649736 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:29.649767 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:29.649792 15996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:00:29.651150 15996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:29.651326 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:29.651501 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:29.656416 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.656394 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.656394 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:29.656792 15996 server_base.cc:1034] running on GCE node
I20260928 15:00:29.657014 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:29.657219 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:29.658401 15996 hybrid_clock.cc:648] HybridClock initialized: now 1790607629658384 us; error 25 us; skew 500 ppm
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:29.661172 15996 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:00:29.661638 15996 webserver.cc:460] Webserver started at http://127.1.252.194:39005/ using document root <none> and password file <none>
I20260928 15:00:29.661871 15996 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:29.661944 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:29.662111 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:29.663389 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "b2b1db46cbcf4253b75dce9372339386"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "d6f5ae124db2b5f9142e35bcce06a45d"
server_key_iv: "a49ac9c9ba06375647a9cc296d130e81"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.663715 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "b2b1db46cbcf4253b75dce9372339386"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "d6f5ae124db2b5f9142e35bcce06a45d"
server_key_iv: "a49ac9c9ba06375647a9cc296d130e81"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.665218 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:00:29.666020 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.666272 15996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:29.666369 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "b2b1db46cbcf4253b75dce9372339386"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "d6f5ae124db2b5f9142e35bcce06a45d"
server_key_iv: "a49ac9c9ba06375647a9cc296d130e81"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.666429 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:29.682102 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:29.684634 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:29.684759 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:29.684948 15996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:29.685287 15996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:29.685325 15996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.685393 15996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:29.685423 15996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.692343 15996 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42593
I20260928 15:00:29.692533 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42593 every 8 connection(s)
I20260928 15:00:29.692800 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:29.693487 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 15996
I20260928 15:00:29.693590 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:00:29.693817 2035 external_mini_cluster.cc:1428] Setting key fcdf843867989fd33e041f96e42c8e77
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
I20260928 15:00:29.704445 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.693602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:33565 (local address 127.1.252.254:45215)
0928 15:00:29.693708 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.693711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.693924 (+ 213us) server_negotiation.cc:407] Connection header received
0928 15:00:29.694627 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.694628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.694640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.694687 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:29.695719 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.696476 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.698222 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.698523 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.698606 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.701299 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:29.701301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:29.701301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:29.701312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:29.702831 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.703363 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.703364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.703365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.703426 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.703689 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.703690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.703690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.703802 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:29.703893 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.704247 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.704348 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:00:29.704699 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.693596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:33565)
0928 15:00:29.693769 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.693815 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.694503 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.694749 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.694753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.695357 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:29.695560 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.695564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.696571 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.696573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.698105 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.698109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.698646 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.698648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.698825 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.699383 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:29.699396 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:29.701177 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:29.702947 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:29.702955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:29.702958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:29.703285 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:29.703497 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:29.703498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:29.703498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:29.703621 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:29.703938 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:29.703945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:29.704084 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.704233 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.704407 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260928 15:00:29.705157 16125 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45215
I20260928 15:00:29.705255 16125 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:29.705406 16125 heartbeater.cc:511] Master 127.1.252.254:45215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
I20260928 15:00:29.705806 15799 ts_manager.cc:194] Registered new tserver with Master: b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593)
I20260928 15:00:29.706249 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:33565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 15:00:29.722539 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45215
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:29.802006 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:29.802204 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:29.802248 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:29.803392 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:29.803434 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:29.803484 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:00:29.804984 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:29.805224 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:29.805444 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:29.810225 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.810225 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:29.810279 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:29.810619 16130 server_base.cc:1034] running on GCE node
I20260928 15:00:29.810842 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:29.811048 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:29.812187 16130 hybrid_clock.cc:648] HybridClock initialized: now 1790607629812176 us; error 32 us; skew 500 ppm
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:29.814985 16130 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:00:29.815435 16130 webserver.cc:460] Webserver started at http://127.1.252.195:40435/ using document root <none> and password file <none>
I20260928 15:00:29.815673 16130 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:29.815748 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:29.815966 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:29.817157 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "1ef1f50176e545acab7e5d6a5d6b06ca"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "40f0ba33dcac5f5a06834f0e51bb0a25"
server_key_iv: "d0e82fe4137c57e53f79462d0a70fd06"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.817543 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "1ef1f50176e545acab7e5d6a5d6b06ca"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "40f0ba33dcac5f5a06834f0e51bb0a25"
server_key_iv: "d0e82fe4137c57e53f79462d0a70fd06"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.818972 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:00:29.819725 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.819940 16130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:29.820044 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "1ef1f50176e545acab7e5d6a5d6b06ca"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "40f0ba33dcac5f5a06834f0e51bb0a25"
server_key_iv: "d0e82fe4137c57e53f79462d0a70fd06"
server_key_version: "encryptionkey@0"
I20260928 15:00:29.820124 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:29.844501 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:29.846971 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:29.847141 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:29.847397 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:29.847715 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:29.847769 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.847810 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:29.847837 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:29.853693 16130 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:44541
I20260928 15:00:29.853751 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:44541 every 8 connection(s)
I20260928 15:00:29.854084 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:29.857203 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16130
I20260928 15:00:29.857318 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:00:29.857479 2035 external_mini_cluster.cc:1428] Setting key 6ada9019f68675702ca965247b91200f
Sep 28 15:00:29 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:29.863621 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.854833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:50735 (local address 127.1.252.254:45215)
0928 15:00:29.854892 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.854894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.855160 (+ 266us) server_negotiation.cc:407] Connection header received
0928 15:00:29.855647 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.855648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.855658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.855709 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:29.856676 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.857185 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.858600 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.858800 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.858855 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.860974 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:29.860976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:29.860976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:29.860983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:29.862427 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.862776 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.862776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.862777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.862824 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.863033 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.863034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.863034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.863130 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:29.863179 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.863468 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.863525 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 15:00:29.863799 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.854829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:50735)
0928 15:00:29.855023 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.855087 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.855574 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.855762 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.855765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.856389 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:29.856583 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.856588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.857280 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.857282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.858536 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.858538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.858871 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.858872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.858967 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.859474 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:29.859482 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:29.860893 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:29.862494 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:29.862500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:29.862501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:29.862711 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:29.862883 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:29.862883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:29.862883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:29.862978 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:29.863220 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:29.863226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:29.863348 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.863445 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.863567 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260928 15:00:29.864382 16259 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45215
I20260928 15:00:29.864490 16259 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:29.864704 16259 heartbeater.cc:511] Master 127.1.252.254:45215 requested a full tablet report, sending...
I20260928 15:00:29.865125 15799 ts_manager.cc:194] Registered new tserver with Master: 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
I20260928 15:00:29.865516 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:50735
I20260928 15:00:29.869169 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:29.877084 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.870618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.1.252.254:45215)
0928 15:00:29.870745 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.870747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.870755 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:00:29.870772 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.870772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.870781 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.870831 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:29.871394 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.871871 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.872732 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.872931 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.873004 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.874202 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:29.874204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:29.874204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:29.874211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:29.875588 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.876071 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.876072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.876073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.876149 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:29.876409 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:29.876410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:29.876410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:29.876667 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:29.876776 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.876946 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.877022 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 15:00:29.878598 15799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:00:29.880210 15799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:00:29.886755 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.882473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:33505 (local address 127.0.0.1:44482)
0928 15:00:29.882778 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.882817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.882890 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.883298 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.883299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.883301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:29.883372 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.883374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.884722 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.884723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.885669 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.885671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.886434 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.886435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.886514 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.886526 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.886604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.886659 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260928 15:00:29.886968 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.882585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.1.252.193:33505)
0928 15:00:29.882765 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.882770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.882983 (+ 213us) server_negotiation.cc:407] Connection header received
0928 15:00:29.883031 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.883034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.883051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.883207 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:29.883457 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.884596 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.885766 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.886336 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.886431 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.886494 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.886825 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.886879 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260928 15:00:29.887015 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.882692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:44541 (local address 127.0.0.1:48180)
0928 15:00:29.882890 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.882898 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.882939 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.883301 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.883302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.883303 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:29.883373 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.883376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.884707 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.884708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.885796 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.885798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.886615 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.886616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.886674 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.886677 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.886730 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.886764 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":34,"thread_start_us":65,"threads_started":1}
I20260928 15:00:29.887147 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.882507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42593 (local address 127.0.0.1:39818)
0928 15:00:29.882831 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.882843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.882900 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.883295 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.883298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.883301 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:29.883366 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.883368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.884330 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.884331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.885549 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.885551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.886574 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.886575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.886649 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.886653 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.886702 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.886748 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":76,"threads_started":1}
I20260928 15:00:29.887209 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.882662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.1.252.194:42593)
0928 15:00:29.883041 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.883046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.883062 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:00:29.883093 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.883094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.883136 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.883211 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:29.883443 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.884233 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.885634 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.886410 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.886481 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.886535 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.886819 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.886961 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":88,"threads_started":1}
I20260928 15:00:29.887748 15924 tablet_service.cc:1467] Processing CreateTablet for tablet 7411a54567dc433ab44804daebbeb213 (DEFAULT_TABLE table=test-table [id=3c62d601ca714d9c962bb7600937e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:29.887943 16059 tablet_service.cc:1467] Processing CreateTablet for tablet 7411a54567dc433ab44804daebbeb213 (DEFAULT_TABLE table=test-table [id=3c62d601ca714d9c962bb7600937e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:29.888164 15924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7411a54567dc433ab44804daebbeb213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:29.886864 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.882886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.1.252.195:44541)
0928 15:00:29.883069 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.883076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.883084 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:00:29.883119 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.883121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.883136 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.883218 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:29.883457 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.884596 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.885876 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.886535 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.886605 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.886668 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.886741 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.886782 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260928 15:00:29.888317 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7411a54567dc433ab44804daebbeb213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:29.890789 16276 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap starting.
I20260928 15:00:29.891155 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 7411a54567dc433ab44804daebbeb213 (DEFAULT_TABLE table=test-table [id=3c62d601ca714d9c962bb7600937e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:29.891434 16276 tablet_bootstrap.cc:654] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:29.891446 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7411a54567dc433ab44804daebbeb213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:29.891963 16276 log.cc:824] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:29.892056 16277 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Bootstrap starting.
I20260928 15:00:29.892652 16276 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: No bootstrap required, opened a new log
I20260928 15:00:29.892705 16277 tablet_bootstrap.cc:654] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:29.892733 16276 ts_tablet_manager.cc:1397] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:00:29.893297 16277 log.cc:824] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:29.894413 16277 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: No bootstrap required, opened a new log
I20260928 15:00:29.894339 16276 raft_consensus.cc:348] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.894487 16277 ts_tablet_manager.cc:1397] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:00:29.894508 16276 raft_consensus.cc:374] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:29.894550 16276 raft_consensus.cc:729] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c51fb5b2c049f69ed4f44848f48aab, State: Initialized, Role: FOLLOWER
I20260928 15:00:29.894661 16276 consensus_queue.cc:260] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.894929 16276 ts_tablet_manager.cc:1428] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:00:29.894908 16280 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Bootstrap starting.
I20260928 15:00:29.895048 15991 heartbeater.cc:503] Master 127.1.252.254:45215 was elected leader, sending a full tablet report...
I20260928 15:00:29.895524 16280 tablet_bootstrap.cc:654] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:29.896116 16280 log.cc:824] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:29.896605 16277 raft_consensus.cc:348] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.896734 16277 raft_consensus.cc:374] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:29.896756 16277 raft_consensus.cc:729] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b1db46cbcf4253b75dce9372339386, State: Initialized, Role: FOLLOWER
I20260928 15:00:29.896905 16280 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: No bootstrap required, opened a new log
I20260928 15:00:29.896881 16277 consensus_queue.cc:260] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.896982 16280 ts_tablet_manager.cc:1397] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:00:29.897176 16125 heartbeater.cc:503] Master 127.1.252.254:45215 was elected leader, sending a full tablet report...
I20260928 15:00:29.897189 16277 ts_tablet_manager.cc:1428] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:29.898689 16280 raft_consensus.cc:348] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.898815 16280 raft_consensus.cc:374] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:29.898837 16280 raft_consensus.cc:729] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef1f50176e545acab7e5d6a5d6b06ca, State: Initialized, Role: FOLLOWER
I20260928 15:00:29.898907 16280 consensus_queue.cc:260] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.899099 16280 ts_tablet_manager.cc:1428] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:00:29.899209 16259 heartbeater.cc:503] Master 127.1.252.254:45215 was elected leader, sending a full tablet report...
I20260928 15:00:29.918342 16284 raft_consensus.cc:482] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:29.918462 16284 raft_consensus.cc:504] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.918907 16284 leader_election.cc:288] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505), b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593)
I20260928 15:00:29.922009 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.919157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:37981 (local address 127.1.252.194:42593)
0928 15:00:29.919202 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.919204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.919214 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:00:29.919308 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.919309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.919323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.919404 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:29.919583 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.920201 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.921160 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.921592 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.921649 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.921655 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.921875 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.921924 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:00:29.922040 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.919084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42593 (local address 127.1.252.195:37981)
0928 15:00:29.919141 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.919153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.919197 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.919415 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.919416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.919418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:29.919499 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.919501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.920280 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.920281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.921059 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.921061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.921710 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.921711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.921798 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.921801 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.921858 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.921921 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:00:29.922039 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.919206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:47313 (local address 127.1.252.193:33505)
0928 15:00:29.919284 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.919286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.919403 (+ 117us) server_negotiation.cc:407] Connection header received
0928 15:00:29.919459 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.919460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.919472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.919531 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:29.919731 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.920301 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.921283 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.921681 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.921756 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.921765 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.921934 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.921982 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:00:29.922254 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.919097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:33505 (local address 127.1.252.195:47313)
0928 15:00:29.919322 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.919333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:29.919393 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:29.919578 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:29.919579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:29.919581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:29.919655 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.919656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.920393 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.920394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.921179 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:29.921181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.921763 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.921764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:29.921818 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.921821 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:00:29.921901 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:00:29.921942 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260928 15:00:29.922570 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7411a54567dc433ab44804daebbeb213" candidate_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" is_pre_election: true
I20260928 15:00:29.922586 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7411a54567dc433ab44804daebbeb213" candidate_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b1db46cbcf4253b75dce9372339386" is_pre_election: true
I20260928 15:00:29.922730 15944 raft_consensus.cc:2454] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ef1f50176e545acab7e5d6a5d6b06ca in term 0.
I20260928 15:00:29.922746 16079 raft_consensus.cc:2454] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ef1f50176e545acab7e5d6a5d6b06ca in term 0.
I20260928 15:00:29.922977 16149 leader_election.cc:302] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef1f50176e545acab7e5d6a5d6b06ca, e1c51fb5b2c049f69ed4f44848f48aab; no voters:
I20260928 15:00:29.923130 16284 raft_consensus.cc:2781] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:29.923189 16284 raft_consensus.cc:482] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:29.923218 16284 raft_consensus.cc:3037] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:29.923979 16284 raft_consensus.cc:504] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.924127 16284 leader_election.cc:288] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [CANDIDATE]: Term 1 election: Requested vote from peers e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505), b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593)
I20260928 15:00:29.924319 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7411a54567dc433ab44804daebbeb213" candidate_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
I20260928 15:00:29.924331 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7411a54567dc433ab44804daebbeb213" candidate_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b1db46cbcf4253b75dce9372339386"
I20260928 15:00:29.924407 15944 raft_consensus.cc:3037] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:29.924431 16079 raft_consensus.cc:3037] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:29.925151 15944 raft_consensus.cc:2454] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ef1f50176e545acab7e5d6a5d6b06ca in term 1.
I20260928 15:00:29.925151 16079 raft_consensus.cc:2454] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ef1f50176e545acab7e5d6a5d6b06ca in term 1.
I20260928 15:00:29.925305 16149 leader_election.cc:302] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef1f50176e545acab7e5d6a5d6b06ca, e1c51fb5b2c049f69ed4f44848f48aab; no voters:
I20260928 15:00:29.925382 16284 raft_consensus.cc:2781] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:29.925547 16284 raft_consensus.cc:686] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 LEADER]: Becoming Leader. State: Replica: 1ef1f50176e545acab7e5d6a5d6b06ca, State: Running, Role: LEADER
I20260928 15:00:29.925654 16284 consensus_queue.cc:237] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:29.926528 15799 catalog_manager.cc:5495] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195). New cstate: current_term: 1 leader_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } health_report { overall_health: UNKNOWN } } }
I20260928 15:00:29.931862 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.929536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.1.252.195:44541)
0928 15:00:29.929584 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:29.929586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:29.929595 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:00:29.929700 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:29.929701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:29.929725 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:29.929802 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:29.929932 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.930380 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.931111 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:29.931316 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:29.931391 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:29.931532 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:29.931631 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:29.931731 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:29.931750 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 15:00:29.931769 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:00:29.933447 15944 raft_consensus.cc:1261] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Refusing update from remote peer 1ef1f50176e545acab7e5d6a5d6b06ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:29.933735 16079 raft_consensus.cc:1261] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Refusing update from remote peer 1ef1f50176e545acab7e5d6a5d6b06ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:29.933759 16286 consensus_queue.cc:1035] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:29.933947 16286 consensus_queue.cc:1035] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:29.936094 16289 mvcc.cc:205] Tried to move back new op lower bound from 7334328852205092864 to 7334328852176310272. Current Snapshot: MvccSnapshot[applied={T|T < 7334328852205092864}]
I20260928 15:00:29.936270 16291 mvcc.cc:205] Tried to move back new op lower bound from 7334328852205092864 to 7334328852176310272. Current Snapshot: MvccSnapshot[applied={T|T < 7334328852205092864}]
I20260928 15:00:29.937060 16290 mvcc.cc:205] Tried to move back new op lower bound from 7334328852205092864 to 7334328852176310272. Current Snapshot: MvccSnapshot[applied={T|T < 7334328852205092864}]
I20260928 15:00:29.938413 15799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:29.938512 15799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:29.939337 15799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Sending DeleteTablet for 3 replicas of tablet 7411a54567dc433ab44804daebbeb213
I20260928 15:00:29.939690 15924 tablet_service.cc:1514] Processing DeleteTablet for tablet 7411a54567dc433ab44804daebbeb213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:29 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:44482
I20260928 15:00:29.939706 16059 tablet_service.cc:1514] Processing DeleteTablet for tablet 7411a54567dc433ab44804daebbeb213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:29 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:39818
I20260928 15:00:29.939780 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15861
I20260928 15:00:29.939944 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 7411a54567dc433ab44804daebbeb213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:29 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:48180
I20260928 15:00:29.940469 16300 tablet_replica.cc:330] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: stopping tablet replica
I20260928 15:00:29.940569 16300 raft_consensus.cc:2229] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:29.940689 16300 raft_consensus.cc:2258] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:29.941162 16300 ts_tablet_manager.cc:1905] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:29.942873 16300 ts_tablet_manager.cc:1918] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:29.942945 16300 log.cc:1196] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/7411a54567dc433ab44804daebbeb213
I20260928 15:00:29.943235 16300 ts_tablet_manager.cc:1939] T 7411a54567dc433ab44804daebbeb213 P b2b1db46cbcf4253b75dce9372339386: Deleting consensus metadata
I20260928 15:00:29.943643 15783 catalog_manager.cc:4841] TS b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): tablet 7411a54567dc433ab44804daebbeb213 (table test-table [id=3c62d601ca714d9c962bb7600937e2ea]) successfully deleted
I20260928 15:00:29.944032 16301 tablet_replica.cc:330] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: stopping tablet replica
I20260928 15:00:29.944154 16301 raft_consensus.cc:2229] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:00:29.944293 16301 raft_consensus.cc:2258] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:29.944802 16301 ts_tablet_manager.cc:1905] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:29.947190 16301 ts_tablet_manager.cc:1918] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:29.947281 16301 log.cc:1196] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/7411a54567dc433ab44804daebbeb213
W20260928 15:00:29.947461 15786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:33505 (error 108)
I20260928 15:00:29.947647 16301 ts_tablet_manager.cc:1939] T 7411a54567dc433ab44804daebbeb213 P 1ef1f50176e545acab7e5d6a5d6b06ca: Deleting consensus metadata
I20260928 15:00:29.948144 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:29.948003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:33505 (local address 127.0.0.1:44498)
0928 15:00:29.948052 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:29.948098 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 15:00:29.948164 15786 catalog_manager.cc:4841] TS 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): tablet 7411a54567dc433ab44804daebbeb213 (table test-table [id=3c62d601ca714d9c962bb7600937e2ea]) successfully deleted
W20260928 15:00:29.948276 15786 catalog_manager.cc:4584] TS e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7411a54567dc433ab44804daebbeb213: Network error: Client connection negotiation failed: client connection to 127.1.252.193:33505: connect: Connection refused (error 111)
I20260928 15:00:29.948391 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15996
I20260928 15:00:29.954434 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 16130
I20260928 15:00:29.960372 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 15767
I20260928 15:00:29.966634 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45215
--webserver_interface=127.1.252.254
--webserver_port=42493
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:30.047827 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.048058 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.048092 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.049208 16303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:30.049268 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.049292 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.049314 16303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:30.049343 16303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:30.050812 16303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:30.051079 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.051308 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.061928 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.061969 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.061969 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.062250 16303 server_base.cc:1034] running on GCE node
I20260928 15:00:30.062505 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.062811 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.063956 16303 hybrid_clock.cc:648] HybridClock initialized: now 1790607630063929 us; error 47 us; skew 500 ppm
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.066432 16303 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:30.066949 16303 webserver.cc:460] Webserver started at http://127.1.252.254:42493/ using document root <none> and password file <none>
I20260928 15:00:30.067201 16303 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.067286 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.068732 16303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.069432 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.069653 16303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.069769 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "f2b67b8bca224f34a7d27e6534eb2492"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "43487e1544fda9e435fbd4b9113b55af"
server_key_iv: "d96e62a6748672b3ee528392471819d6"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.070161 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.092698 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.094651 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.094835 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.099165 16303 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:45215
I20260928 15:00:30.099229 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:45215 every 8 connection(s)
I20260928 15:00:30.099632 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:30.101392 16372 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:00:30.101810 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16303
I20260928 15:00:30.102084 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:33505
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:45215
--webserver_port=41601
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:30.102173 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap starting.
I20260928 15:00:30.111452 16372 log.cc:824] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:30.113701 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:30.114084 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap complete.
I20260928 15:00:30.116781 16372 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:30.117059 16372 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b67b8bca224f34a7d27e6534eb2492, State: Initialized, Role: FOLLOWER
I20260928 15:00:30.117260 16372 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:30.117329 16372 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:30.117368 16372 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:30.117400 16372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:00:30.118521 16372 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:30.118618 16372 leader_election.cc:302] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b67b8bca224f34a7d27e6534eb2492; no voters:
I20260928 15:00:30.118867 16372 leader_election.cc:288] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:00:30.118908 16376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:00:30.119091 16376 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 LEADER]: Becoming Leader. State: Replica: f2b67b8bca224f34a7d27e6534eb2492, State: Running, Role: LEADER
I20260928 15:00:30.119204 16372 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:30.119262 16376 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:30.119588 16376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b67b8bca224f34a7d27e6534eb2492. Latest consensus state: current_term: 2 leader_uuid: "f2b67b8bca224f34a7d27e6534eb2492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } } }
I20260928 15:00:30.119695 16376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:30.119587 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f2b67b8bca224f34a7d27e6534eb2492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } } }
I20260928 15:00:30.120082 16381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:30.120162 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:30.121109 16381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:30.121270 16381 catalog_manager.cc:1259] Loaded cluster ID: 6e9b0a08eab94df598dff6046d9ead59
I20260928 15:00:30.121603 16381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:00:30.121896 16381 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 15:00:30.126346 16381 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 15:00:30.128924 16381 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:30.129130 16381 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Loaded TSK: 0
I20260928 15:00:30.129390 16381 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:00:30.186281 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.186509 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.186546 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.187668 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.187707 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.187732 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:00:30.189172 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:33505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=41601
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:30.189399 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.189646 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.194794 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.194785 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.194787 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.195130 16374 server_base.cc:1034] running on GCE node
I20260928 15:00:30.195356 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.195611 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.196776 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790607630196759 us; error 48 us; skew 500 ppm
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.199424 16374 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:00:30.199833 16374 webserver.cc:460] Webserver started at http://127.1.252.193:41601/ using document root <none> and password file <none>
I20260928 15:00:30.200091 16374 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.200178 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.201606 16374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.202394 16409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.202664 16374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.202768 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "227bc748d83181e949d7c659105e9b66"
server_key_iv: "211748bb5b9f07ab3db084f69618ed59"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.203094 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.213675 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.215968 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.216122 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.216337 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:30.216760 16416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:00:30.218930 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:00:30.218990 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:00:30.219034 16374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:00:30.219720 16374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:00:30.219828 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.220002 16416 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap starting.
I20260928 15:00:30.227196 16374 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:33505
I20260928 15:00:30.227559 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:30.232177 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:33505 every 8 connection(s)
I20260928 15:00:30.232491 16416 log.cc:824] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:30.235556 16416 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:30.236042 16416 tablet_bootstrap.cc:492] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap complete.
I20260928 15:00:30.237344 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16374
I20260928 15:00:30.237576 16416 ts_tablet_manager.cc:1397] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260928 15:00:30.237633 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:42593
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:45215
--webserver_port=39005
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:30.239221 16416 raft_consensus.cc:348] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:30.239759 16416 raft_consensus.cc:729] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c51fb5b2c049f69ed4f44848f48aab, State: Initialized, Role: FOLLOWER
I20260928 15:00:30.240026 16416 consensus_queue.cc:260] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } }
I20260928 15:00:30.240422 16416 ts_tablet_manager.cc:1428] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:00:30.243559 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.228391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:41181 (local address 127.1.252.254:45215)
0928 15:00:30.232081 (+ 3690us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.232092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.232143 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:00:30.232285 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.232292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.232351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.232493 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:30.233289 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.234158 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.235187 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.235566 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.235714 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.239767 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:30.239772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:30.239772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:30.239786 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:30.241665 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.242262 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.242264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.242265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.242363 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.242686 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.242686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.242687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.242886 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:30.242943 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.243315 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.243415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3630,"thread_start_us":108,"threads_started":1}
I20260928 15:00:30.243681 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.228286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:41181)
0928 15:00:30.228473 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.228514 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.228941 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.232523 (+ 3582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.232527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.232909 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:30.233166 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.233171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.234292 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.234293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.235074 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.235076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.235703 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.235704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.235849 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.236463 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:30.236472 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:30.239619 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:30.241771 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.241778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.241780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.242139 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.242441 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.242442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.242442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.242603 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.242958 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:30.242965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:30.243134 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.243263 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.243450 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260928 15:00:30.244534 16524 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45215
I20260928 15:00:30.244640 16524 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:30.244836 16524 heartbeater.cc:511] Master 127.1.252.254:45215 requested a full tablet report, sending...
I20260928 15:00:30.245663 16336 ts_manager.cc:194] Registered new tserver with Master: e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505)
I20260928 15:00:30.247040 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:41181
I20260928 15:00:30.248168 16524 heartbeater.cc:503] Master 127.1.252.254:45215 was elected leader, sending a full tablet report...
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.193@KRBTEST.COM
I20260928 15:00:30.258198 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.246445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:33505 (local address 127.0.0.1:44510)
0928 15:00:30.247387 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.247423 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.247501 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.247921 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.247928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.248336 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:30.248427 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.248429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.250010 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.250012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.251087 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.251091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.251866 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.251867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.251997 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.252628 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:30.252634 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:30.254427 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:30.256855 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.256862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.256864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.257122 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.257395 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.257395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.257396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.257461 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.257811 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:30.257818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:30.257871 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.257956 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.258071 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":116,"threads_started":1}
I20260928 15:00:30.258258 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.246717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.1.252.193:33505)
0928 15:00:30.246911 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.246915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.247591 (+ 676us) server_negotiation.cc:407] Connection header received
0928 15:00:30.247633 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.247635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.247650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.247875 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:30.248731 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.249897 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.251223 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.251757 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.251841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.254585 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:30.254594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:30.254594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:30.254603 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:30.256759 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.257226 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.257229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.257230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.257307 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.257538 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.257539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.257539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.257725 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:30.257828 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.258023 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.258125 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260928 15:00:30.259074 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 7411a54567dc433ab44804daebbeb213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:29 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:44510
I20260928 15:00:30.259732 16533 tablet_replica.cc:330] stopping tablet replica
I20260928 15:00:30.259832 16533 raft_consensus.cc:2229] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:30.259938 16533 raft_consensus.cc:2258] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:30.260286 16533 ts_tablet_manager.cc:1905] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:30.262851 16533 ts_tablet_manager.cc:1918] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:30.262928 16533 log.cc:1196] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/7411a54567dc433ab44804daebbeb213
I20260928 15:00:30.263152 16533 ts_tablet_manager.cc:1939] T 7411a54567dc433ab44804daebbeb213 P e1c51fb5b2c049f69ed4f44848f48aab: Deleting consensus metadata
I20260928 15:00:30.263705 16323 catalog_manager.cc:4841] TS e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505): tablet 7411a54567dc433ab44804daebbeb213 (table test-table [id=3c62d601ca714d9c962bb7600937e2ea]) successfully deleted
W20260928 15:00:30.328871 16529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.329072 16529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.329095 16529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.330238 16529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.330281 16529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.330308 16529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:00:30.331760 16529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:42593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=39005
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:30.332036 16529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.332270 16529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.337306 16539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.337410 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.337320 16538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.337527 16529 server_base.cc:1034] running on GCE node
I20260928 15:00:30.337746 16529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.337951 16529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.339089 16529 hybrid_clock.cc:648] HybridClock initialized: now 1790607630339077 us; error 29 us; skew 500 ppm
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.352381 16529 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:00:30.352830 16529 webserver.cc:460] Webserver started at http://127.1.252.194:39005/ using document root <none> and password file <none>
I20260928 15:00:30.353019 16529 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.353070 16529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.354377 16529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.354928 16548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.355096 16529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.355201 16529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "b2b1db46cbcf4253b75dce9372339386"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "d6f5ae124db2b5f9142e35bcce06a45d"
server_key_iv: "a49ac9c9ba06375647a9cc296d130e81"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.355566 16529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.375650 16529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.378190 16529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.378351 16529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.378607 16529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:30.378950 16529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:30.379009 16529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.379050 16529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:30.379077 16529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.385221 16529 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42593
I20260928 15:00:30.385272 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42593 every 8 connection(s)
I20260928 15:00:30.385614 16529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:30.387444 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16529
I20260928 15:00:30.387699 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:44541
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:45215
--webserver_port=40435
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:30.396013 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.386276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:38859 (local address 127.1.252.254:45215)
0928 15:00:30.386358 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.386360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.386617 (+ 257us) server_negotiation.cc:407] Connection header received
0928 15:00:30.387079 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.387080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.387093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.387146 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:30.388279 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.388846 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.389862 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.390234 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.390323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.393057 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:30.393058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:30.393058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:30.393066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:30.394475 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.394978 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.394979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.394979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.395031 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.395326 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.395327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.395327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.395427 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:30.395475 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.395822 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.395945 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:00:30.396379 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.386262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:38859)
0928 15:00:30.386481 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.386525 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.386997 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.387216 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.387219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.387952 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:30.388173 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.388178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.388956 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.388957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.389775 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.389778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.390334 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.390335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.390436 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.391094 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:30.391126 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:30.392952 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:30.394559 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.394568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.394570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.394897 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.395099 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.395100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.395100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.395256 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.395494 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:30.395500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:30.395680 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.395807 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.396015 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260928 15:00:30.396944 16662 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45215
I20260928 15:00:30.397050 16662 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:30.397262 16662 heartbeater.cc:511] Master 127.1.252.254:45215 requested a full tablet report, sending...
I20260928 15:00:30.397658 16336 ts_manager.cc:194] Registered new tserver with Master: b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593)
I20260928 15:00:30.398039 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:38859
W20260928 15:00:30.472580 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.472774 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.472800 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.474063 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.474105 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.474153 16665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:00:30.475630 16665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:44541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=40435
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:30.475871 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.476151 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.481232 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.481201 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.481189 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.481266 16665 server_base.cc:1034] running on GCE node
I20260928 15:00:30.481611 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.481859 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.483021 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790607630483008 us; error 27 us; skew 500 ppm
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.485457 16665 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:00:30.485884 16665 webserver.cc:460] Webserver started at http://127.1.252.195:40435/ using document root <none> and password file <none>
I20260928 15:00:30.486071 16665 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.486121 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.487336 16665 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:30.487864 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.488080 16665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:30.488162 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "1ef1f50176e545acab7e5d6a5d6b06ca"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "40f0ba33dcac5f5a06834f0e51bb0a25"
server_key_iv: "d0e82fe4137c57e53f79462d0a70fd06"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.488468 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.500320 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.502107 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.502215 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.502391 16665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:30.502740 16665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:30.502775 16665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.502797 16665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:30.502843 16665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.508703 16665 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:44541
I20260928 15:00:30.508782 16793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:44541 every 8 connection(s)
I20260928 15:00:30.509109 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:30.512844 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16665
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:30.518625 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.509804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:59071)
0928 15:00:30.509954 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.509994 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.510453 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.510648 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.510652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.511187 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:30.511370 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.511375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.512095 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.512096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.513055 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.513057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.513549 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.513550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.513732 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.514210 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:30.514220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:30.515532 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:30.517104 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.517110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.517113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.517330 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.517721 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:30.517722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:30.517722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:30.517854 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:30.518166 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:30.518171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:30.518280 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.518364 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.518476 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260928 15:00:30.518636 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.509952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:59071 (local address 127.1.252.254:45215)
0928 15:00:30.510001 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.510004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.510062 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:00:30.510534 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.510535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.510544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.510599 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:30.511459 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.512021 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.513150 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.513468 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.513532 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.515611 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:30.515612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:30.515612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:30.515620 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:30.517016 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.517594 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.517595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.517595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.517650 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.517944 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.517945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.517946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.518085 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:30.518165 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.518412 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.518522 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:00:30.519157 16794 heartbeater.cc:347] Connected to a master server at 127.1.252.254:45215
I20260928 15:00:30.519260 16794 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:30.519452 16794 heartbeater.cc:511] Master 127.1.252.254:45215 requested a full tablet report, sending...
I20260928 15:00:30.519870 16336 ts_manager.cc:194] Registered new tserver with Master: 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
I20260928 15:00:30.519982 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.513237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.1.252.254:45215)
0928 15:00:30.513386 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.513388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.513399 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:00:30.513437 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.513437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.513449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.513511 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:30.514231 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.514685 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.515462 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.515758 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.515831 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.517139 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:30.517141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:30.517141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:30.517148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:30.518665 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.519055 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.519055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.519056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.519113 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:30.519339 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:30.519340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:30.519340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:30.519487 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:30.519691 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.519736 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.519812 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260928 15:00:30.520409 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:59071
I20260928 15:00:30.520728 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:30.523175 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.521059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.1.252.254:45215)
0928 15:00:30.521098 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.521100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.521173 (+ 73us) server_negotiation.cc:407] Connection header received
0928 15:00:30.521241 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.521241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.521267 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.521316 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:30.521400 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.521826 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.522530 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.522704 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.522766 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.522901 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:30.522980 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:30.523036 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.523068 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.523086 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:00:30.523407 16335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:00:30.524945 16335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:00:30.527155 16458 tablet_service.cc:1467] Processing CreateTablet for tablet 5baaf39a319b47e5beea58a762a10114 (DEFAULT_TABLE table=test-table [id=3f8ed994b3034cdd8bf59104db61feef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:30.527381 16458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5baaf39a319b47e5beea58a762a10114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:30.528899 16416 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap starting.
I20260928 15:00:30.529234 16416 tablet_bootstrap.cc:654] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:30.529811 16416 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: No bootstrap required, opened a new log
I20260928 15:00:30.529878 16416 ts_tablet_manager.cc:1397] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:30.530195 16416 raft_consensus.cc:348] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.530282 16416 raft_consensus.cc:374] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:30.530332 16416 raft_consensus.cc:729] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c51fb5b2c049f69ed4f44848f48aab, State: Initialized, Role: FOLLOWER
I20260928 15:00:30.530375 16416 consensus_queue.cc:260] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.530457 16416 ts_tablet_manager.cc:1428] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:30.530742 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.526685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42593 (local address 127.0.0.1:39828)
0928 15:00:30.526746 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.526758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.526794 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.527157 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.527159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.527160 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:30.527201 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.527202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.528479 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.528479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.529464 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.529467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.530506 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.530507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.530574 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.530596 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.530626 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.530662 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 15:00:30.531114 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.526787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.1.252.194:42593)
0928 15:00:30.526970 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.526975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.526985 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:00:30.527029 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.527031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.527048 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.527274 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:30.527350 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.528377 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.529728 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.530424 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.530906 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.530971 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.530995 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.531035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260928 15:00:30.531864 16596 tablet_service.cc:1467] Processing CreateTablet for tablet 5baaf39a319b47e5beea58a762a10114 (DEFAULT_TABLE table=test-table [id=3f8ed994b3034cdd8bf59104db61feef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:30.532248 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5baaf39a319b47e5beea58a762a10114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:30.532562 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.526998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.1.252.195:44541)
0928 15:00:30.527151 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.527155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.529024 (+ 1869us) server_negotiation.cc:407] Connection header received
0928 15:00:30.529063 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.529065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.529079 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.529174 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:30.529269 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.530292 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.531528 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.532251 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.532325 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.532375 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.532477 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.532505 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 15:00:30.532582 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.526753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:44541 (local address 127.0.0.1:48192)
0928 15:00:30.528906 (+ 2153us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.528918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.528969 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.529170 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.529171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.529173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:30.529214 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.529215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.530409 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.530410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.531465 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.531468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.532332 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.532333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.532397 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.532402 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.532468 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.532507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 15:00:30.533324 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 5baaf39a319b47e5beea58a762a10114 (DEFAULT_TABLE table=test-table [id=3f8ed994b3034cdd8bf59104db61feef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:30.533608 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5baaf39a319b47e5beea58a762a10114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:30.535149 16804 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Bootstrap starting.
I20260928 15:00:30.535709 16804 tablet_bootstrap.cc:654] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:30.536216 16804 log.cc:824] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:30.536917 16804 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: No bootstrap required, opened a new log
I20260928 15:00:30.536896 16806 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Bootstrap starting.
I20260928 15:00:30.537005 16804 ts_tablet_manager.cc:1397] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:00:30.537496 16806 tablet_bootstrap.cc:654] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:30.537985 16806 log.cc:824] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:30.538794 16806 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: No bootstrap required, opened a new log
I20260928 15:00:30.538662 16804 raft_consensus.cc:348] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.538841 16804 raft_consensus.cc:374] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:30.538865 16806 ts_tablet_manager.cc:1397] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:00:30.538878 16804 raft_consensus.cc:729] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b1db46cbcf4253b75dce9372339386, State: Initialized, Role: FOLLOWER
I20260928 15:00:30.538997 16804 consensus_queue.cc:260] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.539273 16804 ts_tablet_manager.cc:1428] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:00:30.539299 16662 heartbeater.cc:503] Master 127.1.252.254:45215 was elected leader, sending a full tablet report...
I20260928 15:00:30.540730 16806 raft_consensus.cc:348] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.540851 16806 raft_consensus.cc:374] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:30.540907 16806 raft_consensus.cc:729] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef1f50176e545acab7e5d6a5d6b06ca, State: Initialized, Role: FOLLOWER
I20260928 15:00:30.541006 16806 consensus_queue.cc:260] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.541249 16806 ts_tablet_manager.cc:1428] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:00:30.541290 16794 heartbeater.cc:503] Master 127.1.252.254:45215 was elected leader, sending a full tablet report...
W20260928 15:00:30.636200 16663 tablet.cc:2369] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:00:30.732784 16525 tablet.cc:2369] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:30.748893 16808 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:30.749024 16808 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.749436 16808 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541), e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505)
I20260928 15:00:30.752689 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.749735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:51845 (local address 127.1.252.195:44541)
0928 15:00:30.749799 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.749802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.749842 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:00:30.749873 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.749874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.749885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.749937 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:30.750152 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.750719 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.751879 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.752333 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.752390 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.752396 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.752573 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.752604 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:00:30.752718 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.749627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:44541 (local address 127.1.252.194:51845)
0928 15:00:30.749717 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.749730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.749778 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.749958 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.749958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.749960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:30.750055 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.750057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.750795 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.750797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.751798 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.751800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.752410 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.752411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.752494 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.752498 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.752559 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.752599 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 15:00:30.752825 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.749838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:54683 (local address 127.1.252.193:33505)
0928 15:00:30.749888 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.749890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.750013 (+ 123us) server_negotiation.cc:407] Connection header received
0928 15:00:30.750054 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.750055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.750079 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.750145 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:30.750322 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.750812 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.752030 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.752496 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.752571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.752645 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.752714 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.752759 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:00:30.753016 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.749762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:33505 (local address 127.1.252.194:54683)
0928 15:00:30.749931 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.749943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:30.750001 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:30.750171 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:30.750172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:30.750174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:30.750248 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.750250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.750893 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.750893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.751942 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:30.751945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.752581 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.752582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:30.752650 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.752654 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:00:30.752709 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 15:00:30.752939 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":10,"thread_start_us":107,"threads_started":1}
I20260928 15:00:30.753278 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baaf39a319b47e5beea58a762a10114" candidate_uuid: "b2b1db46cbcf4253b75dce9372339386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" is_pre_election: true
I20260928 15:00:30.753340 16473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baaf39a319b47e5beea58a762a10114" candidate_uuid: "b2b1db46cbcf4253b75dce9372339386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" is_pre_election: true
I20260928 15:00:30.753446 16748 raft_consensus.cc:2454] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b1db46cbcf4253b75dce9372339386 in term 0.
I20260928 15:00:30.753460 16473 raft_consensus.cc:2454] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b1db46cbcf4253b75dce9372339386 in term 0.
I20260928 15:00:30.753657 16552 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2b1db46cbcf4253b75dce9372339386, e1c51fb5b2c049f69ed4f44848f48aab; no voters:
I20260928 15:00:30.753824 16808 raft_consensus.cc:2781] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:30.753886 16808 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:30.753902 16808 raft_consensus.cc:3037] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:30.754683 16808 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.754809 16808 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [CANDIDATE]: Term 1 election: Requested vote from peers 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541), e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505)
I20260928 15:00:30.754974 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baaf39a319b47e5beea58a762a10114" candidate_uuid: "b2b1db46cbcf4253b75dce9372339386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca"
I20260928 15:00:30.755012 16473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baaf39a319b47e5beea58a762a10114" candidate_uuid: "b2b1db46cbcf4253b75dce9372339386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
I20260928 15:00:30.755054 16748 raft_consensus.cc:3037] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:30.755121 16473 raft_consensus.cc:3037] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:30.755827 16748 raft_consensus.cc:2454] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b1db46cbcf4253b75dce9372339386 in term 1.
I20260928 15:00:30.755849 16473 raft_consensus.cc:2454] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b1db46cbcf4253b75dce9372339386 in term 1.
I20260928 15:00:30.756022 16552 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386; no voters:
I20260928 15:00:30.756124 16808 raft_consensus.cc:2781] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:30.756278 16808 raft_consensus.cc:686] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 1 LEADER]: Becoming Leader. State: Replica: b2b1db46cbcf4253b75dce9372339386, State: Running, Role: LEADER
I20260928 15:00:30.756372 16808 consensus_queue.cc:237] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:30.757133 16335 catalog_manager.cc:5495] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2b1db46cbcf4253b75dce9372339386 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "b2b1db46cbcf4253b75dce9372339386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } health_report { overall_health: UNKNOWN } } }
W20260928 15:00:30.759586 16795 tablet.cc:2369] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:30.811964 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.809791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.1.252.194:42593)
0928 15:00:30.809859 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:30.809861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:30.809869 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:00:30.809907 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:30.809907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:30.809924 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:30.809984 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:30.810092 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.810551 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.811251 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:30.811425 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:30.811473 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:30.811594 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:30.811688 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:30.811751 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:30.811782 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 15:00:30.811800 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 15:00:30.813843 16748 raft_consensus.cc:1261] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Refusing update from remote peer b2b1db46cbcf4253b75dce9372339386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:30.813843 16473 raft_consensus.cc:1261] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Refusing update from remote peer b2b1db46cbcf4253b75dce9372339386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:30.814157 16808 consensus_queue.cc:1035] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:30.814356 16811 consensus_queue.cc:1035] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:30.816172 16814 mvcc.cc:205] Tried to move back new op lower bound from 7334328855810531328 to 7334328855578533888. Current Snapshot: MvccSnapshot[applied={T|T < 7334328855810531328}]
I20260928 15:00:30.816457 16815 mvcc.cc:205] Tried to move back new op lower bound from 7334328855810531328 to 7334328855578533888. Current Snapshot: MvccSnapshot[applied={T|T < 7334328855810531328}]
I20260928 15:00:30.818432 16336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:30.818524 16336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:30.819271 16336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Sending DeleteTablet for 3 replicas of tablet 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:30.819628 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 5baaf39a319b47e5beea58a762a10114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:30 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:44510
I20260928 15:00:30.819666 16596 tablet_service.cc:1514] Processing DeleteTablet for tablet 5baaf39a319b47e5beea58a762a10114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:30 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:39828
I20260928 15:00:30.819823 16824 tablet_replica.cc:330] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: stopping tablet replica
I20260928 15:00:30.819840 16728 tablet_service.cc:1514] Processing DeleteTablet for tablet 5baaf39a319b47e5beea58a762a10114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:30 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:48192
I20260928 15:00:30.819877 16824 raft_consensus.cc:2229] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:30.819957 16824 raft_consensus.cc:2258] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:30.819968 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 16374
I20260928 15:00:30.820674 16825 tablet_replica.cc:330] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: stopping tablet replica
I20260928 15:00:30.820803 16825 raft_consensus.cc:2229] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:00:30.820922 16825 raft_consensus.cc:2258] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:30.821413 16825 ts_tablet_manager.cc:1905] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:30.823087 16825 ts_tablet_manager.cc:1918] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:30.823232 16825 log.cc:1196] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/5baaf39a319b47e5beea58a762a10114
I20260928 15:00:30.823565 16825 ts_tablet_manager.cc:1939] T 5baaf39a319b47e5beea58a762a10114 P b2b1db46cbcf4253b75dce9372339386: Deleting consensus metadata
I20260928 15:00:30.823987 16320 catalog_manager.cc:4841] TS b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): tablet 5baaf39a319b47e5beea58a762a10114 (table test-table [id=3f8ed994b3034cdd8bf59104db61feef]) successfully deleted
W20260928 15:00:30.826087 16323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:33505 (error 108)
I20260928 15:00:30.826417 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 16529
I20260928 15:00:30.826968 16826 tablet_replica.cc:330] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: stopping tablet replica
I20260928 15:00:30.826982 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:30.826501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:33505 (local address 127.0.0.1:44512)
0928 15:00:30.826903 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:30.826946 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
I20260928 15:00:30.827077 16826 raft_consensus.cc:2229] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:30.827184 16826 raft_consensus.cc:2258] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:30.828802 16826 ts_tablet_manager.cc:1905] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:00:30.828588 16323 catalog_manager.cc:4584] TS e1c51fb5b2c049f69ed4f44848f48aab (127.1.252.193:33505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5baaf39a319b47e5beea58a762a10114: Network error: Client connection negotiation failed: client connection to 127.1.252.193:33505: connect: Connection refused (error 111)
I20260928 15:00:30.830281 16826 ts_tablet_manager.cc:1918] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:30.830367 16826 log.cc:1196] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/5baaf39a319b47e5beea58a762a10114
I20260928 15:00:30.830634 16826 ts_tablet_manager.cc:1939] T 5baaf39a319b47e5beea58a762a10114 P 1ef1f50176e545acab7e5d6a5d6b06ca: Deleting consensus metadata
I20260928 15:00:30.831007 16323 catalog_manager.cc:4841] TS 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): tablet 5baaf39a319b47e5beea58a762a10114 (table test-table [id=3f8ed994b3034cdd8bf59104db61feef]) successfully deleted
I20260928 15:00:30.832479 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 16665
I20260928 15:00:30.838305 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 16303
I20260928 15:00:30.843799 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45215
--webserver_interface=127.1.252.254
--webserver_port=42493
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:30.925421 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:30.925623 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:30.925645 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:30.926777 16828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:30.926816 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:30.926831 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:30.926844 16828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:30.926856 16828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:30.928375 16828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:30.928623 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:30.928857 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:30.933727 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.933744 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:30.933808 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:30.933920 16828 server_base.cc:1034] running on GCE node
I20260928 15:00:30.934151 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:30.934448 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:30.935621 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790607630935615 us; error 33 us; skew 500 ppm
Sep 28 15:00:30 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:30.938057 16828 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:30.938498 16828 webserver.cc:460] Webserver started at http://127.1.252.254:42493/ using document root <none> and password file <none>
I20260928 15:00:30.938750 16828 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:30.938796 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:30.940264 16828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:00:30.940914 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:30.941112 16828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:00:30.941202 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "f2b67b8bca224f34a7d27e6534eb2492"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "43487e1544fda9e435fbd4b9113b55af"
server_key_iv: "d96e62a6748672b3ee528392471819d6"
server_key_version: "encryptionkey@0"
I20260928 15:00:30.941540 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:30.982134 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:30.984267 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:30.984411 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:30.988421 16828 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:45215
I20260928 15:00:30.988504 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:45215 every 8 connection(s)
I20260928 15:00:30.988888 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:30.990530 16896 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:00:30.991218 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap starting.
I20260928 15:00:30.999224 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16828
I20260928 15:00:30.999573 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:33505
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:45215
--webserver_port=41601
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:31.000192 16896 log.cc:824] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:31.002517 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:31.002827 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492: Bootstrap complete.
I20260928 15:00:31.005203 16896 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:31.005417 16896 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b67b8bca224f34a7d27e6534eb2492, State: Initialized, Role: FOLLOWER
I20260928 15:00:31.005595 16896 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:31.005656 16896 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:31.005716 16896 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:31.005766 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:00:31.006775 16896 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:31.006860 16896 leader_election.cc:302] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b67b8bca224f34a7d27e6534eb2492; no voters:
I20260928 15:00:31.007061 16896 leader_election.cc:288] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:00:31.007097 16900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:00:31.007201 16900 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [term 3 LEADER]: Becoming Leader. State: Replica: f2b67b8bca224f34a7d27e6534eb2492, State: Running, Role: LEADER
I20260928 15:00:31.007299 16900 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } }
I20260928 15:00:31.007328 16896 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:31.007576 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f2b67b8bca224f34a7d27e6534eb2492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } } }
I20260928 15:00:31.007771 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b67b8bca224f34a7d27e6534eb2492. Latest consensus state: current_term: 3 leader_uuid: "f2b67b8bca224f34a7d27e6534eb2492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b67b8bca224f34a7d27e6534eb2492" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 45215 } } }
I20260928 15:00:31.007982 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:31.007851 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b67b8bca224f34a7d27e6534eb2492 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:31.008225 16910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:31.009084 16910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:31.009202 16910 catalog_manager.cc:1259] Loaded cluster ID: 6e9b0a08eab94df598dff6046d9ead59
I20260928 15:00:31.009276 16910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 15:00:31.009752 16910 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f51e62969ad google::LogMessage::Fail() at ??:0
@ 0x7f51e629988a google::LogMessage::SendToLog() at ??:0
@ 0x7f51e62964de google::LogMessage::Flush() at ??:0
@ 0x7f51e6297a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f51ee1b70f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f51ee1c08b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f51e81640d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f51e8159954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f51e8fbd6db start_thread at ??:0
@ 0x7f51e4d9461f clone at ??:0
W20260928 15:00:31.096356 16899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:31.096539 16899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:31.096562 16899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:31.097725 16899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:31.097767 16899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:31.097793 16899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:00:31.099246 16899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:33505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=41601
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:31.099468 16899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:31.099718 16899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:31.104710 16933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.104738 16932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.104943 16899 server_base.cc:1034] running on GCE node
W20260928 15:00:31.104736 16935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.105194 16899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:31.105474 16899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:31.106660 16899 hybrid_clock.cc:648] HybridClock initialized: now 1790607631106646 us; error 42 us; skew 500 ppm
Sep 28 15:00:31 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:31.109686 16899 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:00:31.110200 16899 webserver.cc:460] Webserver started at http://127.1.252.193:41601/ using document root <none> and password file <none>
I20260928 15:00:31.110468 16899 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:31.110534 16899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:31.112432 16899 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:00:31.113256 16942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.113463 16899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:00:31.113569 16899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "e1c51fb5b2c049f69ed4f44848f48aab"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "227bc748d83181e949d7c659105e9b66"
server_key_iv: "211748bb5b9f07ab3db084f69618ed59"
server_key_version: "encryptionkey@0"
I20260928 15:00:31.114015 16899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:31.129112 16899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:31.131337 16899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:31.131482 16899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:31.131740 16899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:31.132239 16949 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:00:31.133651 16899 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:00:31.133710 16899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:00:31.133744 16899 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:00:31.134536 16899 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:00:31.134589 16899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:31.134649 16949 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap starting.
I20260928 15:00:31.141935 16899 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:33505
I20260928 15:00:31.142429 16899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:00:31.145133 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 16899
I20260928 15:00:31.145408 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:42593
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:45215
--webserver_port=39005
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:00:31.148049 17056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:33505 every 8 connection(s)
I20260928 15:00:31.148573 16949 log.cc:824] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:31.151633 16949 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:00:31.151962 16949 tablet_bootstrap.cc:492] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Bootstrap complete.
I20260928 15:00:31.152947 16949 ts_tablet_manager.cc:1397] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.003s
I20260928 15:00:31.153988 16949 raft_consensus.cc:348] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:31.154217 16949 raft_consensus.cc:729] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c51fb5b2c049f69ed4f44848f48aab, State: Initialized, Role: FOLLOWER
I20260928 15:00:31.154356 16949 consensus_queue.cc:260] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:31.154627 16949 ts_tablet_manager.cc:1428] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 15:00:31.199481 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:00:31.143438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:55561)
0928 15:00:31.143668 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.143761 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:31.144629 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:31.199271 (+ 54642us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":142,"threads_started":1}
W20260928 15:00:31.199847 17057 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:45215 (0 consecutive failures): Network error: Failed to ping master at 127.1.252.254:45215: Client connection negotiation failed: client connection to 127.1.252.254:45215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:00:31.200265 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.200161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:38389)
0928 15:00:31.200201 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.200242 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 15:00:31.200650 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.200545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:44661)
0928 15:00:31.200594 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.200626 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:00:31.200759 17057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:00:31.237174 17060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:31.237390 17060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:31.237427 17060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:31.238536 17060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:31.238605 17060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:31.238651 17060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:00:31.240470 17060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:42593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=39005
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:31.240758 17060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:31.240979 17060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:31.245873 17068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.245900 17070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.246109 17060 server_base.cc:1034] running on GCE node
W20260928 15:00:31.245886 17067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.246358 17060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:31.246618 17060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:31.247790 17060 hybrid_clock.cc:648] HybridClock initialized: now 1790607631247774 us; error 25 us; skew 500 ppm
Sep 28 15:00:31 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:31.250312 17060 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:00:31.250675 17060 webserver.cc:460] Webserver started at http://127.1.252.194:39005/ using document root <none> and password file <none>
I20260928 15:00:31.250905 17060 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:31.250950 17060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:31.252298 17060 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:31.252912 17077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.253091 17060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:00:31.253170 17060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "b2b1db46cbcf4253b75dce9372339386"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "d6f5ae124db2b5f9142e35bcce06a45d"
server_key_iv: "a49ac9c9ba06375647a9cc296d130e81"
server_key_version: "encryptionkey@0"
I20260928 15:00:31.253474 17060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:31.266685 17060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:31.268633 17060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:31.268751 17060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:31.268914 17060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:31.269194 17060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:31.269229 17060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.269250 17060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:31.269263 17060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.274972 17060 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42593
I20260928 15:00:31.275054 17190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42593 every 8 connection(s)
I20260928 15:00:31.275368 17060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:31.276425 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.276070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:40717)
0928 15:00:31.276221 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.276288 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
W20260928 15:00:31.276852 17191 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:45215 (0 consecutive failures): Network error: Failed to ping master at 127.1.252.254:45215: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
I20260928 15:00:31.277173 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.277037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:60381)
0928 15:00:31.277105 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.277134 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 15:00:31.277669 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.277447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:58225)
0928 15:00:31.277489 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.277634 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 15:00:31.277813 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17060
W20260928 15:00:31.277829 17191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:00:31.278056 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:44541
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:45215
--webserver_port=40435
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:31.358886 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:31.359077 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:31.359100 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:31.360172 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:31.360204 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:31.360230 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:00:31.361550 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:44541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=40435
--tserver_master_addrs=127.1.252.254:45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:31.361735 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:31.361899 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:31.367192 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:31.367197 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.367273 17194 server_base.cc:1034] running on GCE node
W20260928 15:00:31.367173 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:31.367642 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:31.367843 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:31.369021 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790607631369009 us; error 27 us; skew 500 ppm
Sep 28 15:00:31 dist-test-slave-sf70 krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:31.371598 17194 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:00:31.372083 17194 webserver.cc:460] Webserver started at http://127.1.252.195:40435/ using document root <none> and password file <none>
I20260928 15:00:31.372280 17194 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:31.372323 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:31.374164 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:00:31.374788 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.374962 17194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:31.375044 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "1ef1f50176e545acab7e5d6a5d6b06ca"
format_stamp: "Formatted at 2026-09-28 15:00:29 on dist-test-slave-sf70"
server_key: "40f0ba33dcac5f5a06834f0e51bb0a25"
server_key_iv: "d0e82fe4137c57e53f79462d0a70fd06"
server_key_version: "encryptionkey@0"
I20260928 15:00:31.375341 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:31.413111 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:31.415139 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:31.415275 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:31.415472 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:31.415807 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:31.415843 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.415865 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:31.415947 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:31.422242 17194 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:44541
I20260928 15:00:31.422298 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:44541 every 8 connection(s)
I20260928 15:00:31.422623 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:00:31.423170 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17194
I20260928 15:00:31.423787 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.423333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:56989)
0928 15:00:31.423511 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.423591 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
W20260928 15:00:31.424221 17323 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:45215 (0 consecutive failures): Network error: Failed to ping master at 127.1.252.254:45215: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
I20260928 15:00:31.424503 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.424408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:53483)
0928 15:00:31.424451 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.424479 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 15:00:31.424849 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:31.424769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:48039)
0928 15:00:31.424811 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:31.424834 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 15:00:31.424983 17323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:00:32.201373 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.201100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:60997)
0928 15:00:32.201275 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.201307 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260928 15:00:32.278347 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.278142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:36195)
0928 15:00:32.278267 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.278299 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":54,"threads_started":1}
I20260928 15:00:32.425521 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.425301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:60605)
0928 15:00:32.425435 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.425471 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260928 15:00:32.450901 17330 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:32.451072 17330 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:32.451435 17330 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593), 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
Sep 28 15:00:32 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.194@KRBTEST.COM
Sep 28 15:00:32 dist-test-slave-sf70 krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.195@KRBTEST.COM
I20260928 15:00:32.464095 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.451650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42593 (local address 127.1.252.193:41067)
0928 15:00:32.451759 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.451771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:32.451827 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:32.452870 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:32.452873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:32.453372 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:32.453652 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.453661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.455370 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.455371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.456654 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.456657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.457500 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.457501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.457660 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.458305 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:32.458316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:32.460163 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:32.462334 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:32.462342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:32.462344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:32.462720 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:32.462985 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:32.462986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:32.462986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:32.463113 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:32.463517 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:32.463524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:32.463679 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 15:00:32.463789 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:00:32.464000 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":118,"thread_start_us":67,"threads_started":1}
I20260928 15:00:32.464108 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.451747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:54599 (local address 127.1.252.195:44541)
0928 15:00:32.451941 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.451953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.451989 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:00:32.452588 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.452591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.452632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.452785 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:32.453809 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.455223 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.456776 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.457391 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.457507 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.460687 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:32.460691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:32.460691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:32.460703 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:32.462516 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.462904 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.462906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.462907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.463033 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.463263 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.463263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.463264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.463493 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:32.463577 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.463847 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.463988 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260928 15:00:32.464145 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.451806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:41067 (local address 127.1.252.194:42593)
0928 15:00:32.451958 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:32.451966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:32.451994 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:00:32.452572 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:32.452575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:32.452632 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:32.452785 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:32.453809 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.455226 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.456765 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.457391 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.457508 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.460281 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:32.460286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:32.460287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:32.460299 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:32.462151 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.462799 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.462803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.462803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.462914 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:32.463181 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:32.463182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:32.463182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:32.463442 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:32.463504 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:32.463857 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:00:32.464031 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260928 15:00:32.464561 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:32.451576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:44541 (local address 127.1.252.193:54599)
0928 15:00:32.451656 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:32.451669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:32.451731 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:32.452870 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:32.452873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:32.453372 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:32.453652 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.453661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.455379 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.455379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.456654 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:32.456657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:32.457503 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:32.457504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:32.457660 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:32.458311 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:32.458320 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:32.460572 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:32.462606 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:32.462607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:32.462608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:32.462822 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:32.463112 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:32.463113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:32.463113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:32.463192 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:32.463576 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:32.463577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:32.463679 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 15:00:32.463791 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 15:00:32.464090 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":152}
W20260928 15:00:32.465215 16943 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
W20260928 15:00:32.465302 16946 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:32.465361 16946 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c51fb5b2c049f69ed4f44848f48aab; no voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386
I20260928 15:00:32.465461 17330 raft_consensus.cc:2726] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:33.202009 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:33.201785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:50087)
0928 15:00:33.201923 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:33.201961 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 15:00:33.278982 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:33.278791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:36751)
0928 15:00:33.278909 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:33.278944 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 15:00:33.426152 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:33.425955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:53127)
0928 15:00:33.426084 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:33.426117 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 15:00:34.202801 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:34.202541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:46885)
0928 15:00:34.202709 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:34.202751 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260928 15:00:34.279947 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:34.279675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:40989)
0928 15:00:34.279815 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:34.279849 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260928 15:00:34.374661 17339 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:34.374752 17339 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:34.374972 17339 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593), 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
W20260928 15:00:34.375517 16946 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
W20260928 15:00:34.375594 16943 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:34.375628 16943 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c51fb5b2c049f69ed4f44848f48aab; no voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386
I20260928 15:00:34.375716 17339 raft_consensus.cc:2726] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:34.426803 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:34.426598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:51201)
0928 15:00:34.426725 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:34.426763 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260928 15:00:35.203491 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:35.203288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:53789)
0928 15:00:35.203409 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:35.203444 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 15:00:35.280608 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:35.280411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:60625)
0928 15:00:35.280533 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:35.280570 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 15:00:35.427524 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:35.427306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:37063)
0928 15:00:35.427436 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:35.427476 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260928 15:00:36.204353 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:36.204078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:37547)
0928 15:00:36.204235 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:36.204272 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260928 15:00:36.281262 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:36.281039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:52647)
0928 15:00:36.281185 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:36.281218 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260928 15:00:36.428237 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:36.427991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:58569)
0928 15:00:36.428162 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:36.428198 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260928 15:00:37.204993 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:37.204792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:42977)
0928 15:00:37.204910 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:37.204945 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260928 15:00:37.281873 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:37.281667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:50197)
0928 15:00:37.281801 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:37.281835 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":54,"threads_started":1}
I20260928 15:00:37.422720 17349 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:37.422827 17349 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:37.423089 17349 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593), 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
W20260928 15:00:37.423353 16946 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
W20260928 15:00:37.423419 16943 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:37.423454 16943 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c51fb5b2c049f69ed4f44848f48aab; no voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386
I20260928 15:00:37.423553 17349 raft_consensus.cc:2726] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:37.428855 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:37.428648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:48127)
0928 15:00:37.428787 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:37.428821 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":102,"threads_started":1}
I20260928 15:00:38.205662 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:38.205431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:54849)
0928 15:00:38.205573 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:38.205613 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 15:00:38.282574 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:38.282318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:52403)
0928 15:00:38.282484 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:38.282529 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260928 15:00:38.429502 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:38.429288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:52265)
0928 15:00:38.429424 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:38.429458 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260928 15:00:39.206369 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:39.206145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:41631)
0928 15:00:39.206284 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:39.206323 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260928 15:00:39.283303 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:39.283093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:56055)
0928 15:00:39.283232 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:39.283262 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 15:00:39.430171 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:39.429976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:53283)
0928 15:00:39.430095 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:39.430130 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260928 15:00:40.207134 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:40.206899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:50661)
0928 15:00:40.207039 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:40.207079 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260928 15:00:40.284199 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:40.283851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:34581)
0928 15:00:40.284112 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:40.284155 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
I20260928 15:00:40.431022 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:40.430731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:36413)
0928 15:00:40.430936 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:40.430975 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260928 15:00:41.091156 17360 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:41.091257 17360 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:41.091427 17360 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593), 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
W20260928 15:00:41.091871 16946 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
W20260928 15:00:41.091980 16943 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:41.092018 16943 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c51fb5b2c049f69ed4f44848f48aab; no voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386
I20260928 15:00:41.092108 17360 raft_consensus.cc:2726] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:41.207856 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:41.207640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:35181)
0928 15:00:41.207776 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:41.207810 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 15:00:41.285029 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:41.284782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:38037)
0928 15:00:41.284941 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:41.284981 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260928 15:00:41.431818 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:41.431589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:56423)
0928 15:00:41.431723 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:41.431759 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260928 15:00:42.208685 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:42.208376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:48745)
0928 15:00:42.208595 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:42.208638 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":62,"threads_started":1}
I20260928 15:00:42.285792 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:42.285587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:58839)
0928 15:00:42.285720 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:42.285751 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 15:00:42.432610 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:42.432399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:38251)
0928 15:00:42.432525 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:42.432562 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260928 15:00:43.209556 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:43.209325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:33579)
0928 15:00:43.209472 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:43.209509 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260928 15:00:43.286577 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:43.286346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:60259)
0928 15:00:43.286494 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:43.286529 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 15:00:43.343364 17369 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:43.343479 17369 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:43.343726 17369 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593), 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
W20260928 15:00:43.344148 16943 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
W20260928 15:00:43.344312 16946 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:43.344393 16946 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c51fb5b2c049f69ed4f44848f48aab; no voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386
I20260928 15:00:43.344462 17369 raft_consensus.cc:2726] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:43.433399 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:43.433166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:59279)
0928 15:00:43.433308 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:43.433353 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
I20260928 15:00:44.210314 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:44.210103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:49313)
0928 15:00:44.210230 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:44.210266 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260928 15:00:44.287348 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:44.287127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:45663)
0928 15:00:44.287257 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:44.287305 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260928 15:00:44.434247 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:44.433985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:54755)
0928 15:00:44.434152 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:44.434196 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260928 15:00:45.211102 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:45.210843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:45831)
0928 15:00:45.211014 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:45.211053 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260928 15:00:45.288309 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:45.288066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:35131)
0928 15:00:45.288227 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:45.288263 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260928 15:00:45.291612 17376 raft_consensus.cc:482] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:45.291719 17376 raft_consensus.cc:504] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b1db46cbcf4253b75dce9372339386" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42593 } } peers { permanent_uuid: "1ef1f50176e545acab7e5d6a5d6b06ca" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 44541 } } peers { permanent_uuid: "e1c51fb5b2c049f69ed4f44848f48aab" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 33505 } }
I20260928 15:00:45.292166 17376 leader_election.cc:288] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593), 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541)
W20260928 15:00:45.292546 16943 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2b1db46cbcf4253b75dce9372339386 (127.1.252.194:42593): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
W20260928 15:00:45.292696 16946 leader_election.cc:341] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef1f50176e545acab7e5d6a5d6b06ca (127.1.252.195:44541): Not found: Tablet not found: 5baaf39a319b47e5beea58a762a10114
I20260928 15:00:45.292757 16946 leader_election.cc:302] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c51fb5b2c049f69ed4f44848f48aab; no voters: 1ef1f50176e545acab7e5d6a5d6b06ca, b2b1db46cbcf4253b75dce9372339386
I20260928 15:00:45.292837 17376 raft_consensus.cc:2726] T 5baaf39a319b47e5beea58a762a10114 P e1c51fb5b2c049f69ed4f44848f48aab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:00:45.435119 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:45.434883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:60717)
0928 15:00:45.435038 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:45.435072 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260928 15:00:46.211838 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:46.211648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.193:43115)
0928 15:00:46.211763 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:46.211793 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 15:00:46.289098 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:46.288863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.194:55235)
0928 15:00:46.289007 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:46.289052 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260928 15:00:46.428777 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 16899
I20260928 15:00:46.433748 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17060
I20260928 15:00:46.435959 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:46.435713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:45215 (local address 127.1.252.195:37231)
0928 15:00:46.435837 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:46.435871 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 15:00:46.438485 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17194
2026-09-28T15:00:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:00:46 dist-test-slave-sf70 krb5kdc[17383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:00:46 dist-test-slave-sf70 krb5kdc[17383](info): set up 2 sockets
Sep 28 15:00:46 dist-test-slave-sf70 krb5kdc[17383](info): commencing operation
krb5kdc: starting...
W20260928 15:00:47.517705 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:00:47 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:00:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:00:47.643743 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:36033
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:47.725725 17396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:47.725922 17396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:47.725944 17396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:47.727093 17396 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:00:47.727144 17396 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:00:47.727159 17396 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:00:47.727169 17396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:00:47.727214 17396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:47.727236 17396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:47.727255 17396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:00:47.727347 17396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:00:47.728916 17396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:47.729110 17396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:47.729281 17396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:47.734302 17402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:47.734367 17396 server_base.cc:1034] running on GCE node
W20260928 15:00:47.734297 17404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:47.734298 17401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:47.734733 17396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:47.735018 17396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:47.736172 17396 hybrid_clock.cc:648] HybridClock initialized: now 1790607647736155 us; error 35 us; skew 500 ppm
Sep 28 15:00:47 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607647, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:47.738660 17396 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:00:47.739113 17396 webserver.cc:460] Webserver started at http://127.1.252.254:37727/ using document root <none> and password file <none>
I20260928 15:00:47.739355 17396 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:47.739427 17396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:47.739584 17396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:47.740710 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "7c41c0c8f9604f75862b9a0ed05017d8"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "41e35cb3f26e30be94235990c77c8e3d"
server_key_iv: "efbf196ff05acf5a234c9f0df313522c"
server_key_version: "encryptionkey@0"
I20260928 15:00:47.741072 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "7c41c0c8f9604f75862b9a0ed05017d8"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "41e35cb3f26e30be94235990c77c8e3d"
server_key_iv: "efbf196ff05acf5a234c9f0df313522c"
server_key_version: "encryptionkey@0"
I20260928 15:00:47.742424 17396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:00:47.743194 17411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:47.743434 17396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:00:47.743543 17396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "7c41c0c8f9604f75862b9a0ed05017d8"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "41e35cb3f26e30be94235990c77c8e3d"
server_key_iv: "efbf196ff05acf5a234c9f0df313522c"
server_key_version: "encryptionkey@0"
I20260928 15:00:47.743628 17396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:47.766958 17396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:47.769503 17396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:47.769682 17396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:47.773784 17396 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:36033
I20260928 15:00:47.773860 17464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:36033 every 8 connection(s)
I20260928 15:00:47.774375 17396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:00:47.775637 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:47.777837 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap starting.
I20260928 15:00:47.778383 17465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:47.778635 17465 log.cc:824] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:47.778716 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17396
I20260928 15:00:47.778811 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:00:47.779030 2035 external_mini_cluster.cc:1428] Setting key 6bc97699d8441a94be0973baed56a417
I20260928 15:00:47.779474 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: No bootstrap required, opened a new log
I20260928 15:00:47.781574 17465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:00:47.781747 17465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:47.781790 17465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c41c0c8f9604f75862b9a0ed05017d8, State: Initialized, Role: FOLLOWER
I20260928 15:00:47.781920 17465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:00:47.782009 17465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:00:47.782050 17465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:00:47.782096 17465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:47.782799 17465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:00:47.782902 17465 leader_election.cc:302] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c41c0c8f9604f75862b9a0ed05017d8; no voters:
I20260928 15:00:47.783046 17465 leader_election.cc:288] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:00:47.783109 17471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:47.783268 17465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:00:47.783284 17471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 LEADER]: Becoming Leader. State: Replica: 7c41c0c8f9604f75862b9a0ed05017d8, State: Running, Role: LEADER
I20260928 15:00:47.783380 17471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:00:47.783767 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c41c0c8f9604f75862b9a0ed05017d8. Latest consensus state: current_term: 1 leader_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } } }
I20260928 15:00:47.783828 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:47.783967 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } } }
I20260928 15:00:47.784040 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:00:47.784786 17483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:00:47.785279 17483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:00:47.786824 17483 catalog_manager.cc:1347] Generated new cluster ID: ff7dff5a5fd04852be0a91d97695c07d
I20260928 15:00:47.786876 17483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:00:47 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:47.792413 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:47.779560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.1.252.254:36033)
0928 15:00:47.779781 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:47.779790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:47.779821 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:00:47.780764 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:47.780773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:47.780832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:47.780980 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:47.781614 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:47.783382 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:47.784502 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:47.785116 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:47.785226 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:47.788185 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:47.788189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:47.788189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:47.788201 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:47.790386 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:47.790794 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:47.790797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:47.790797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:47.790900 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:47.791108 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:47.791109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:47.791109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:47.791429 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:47.791575 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:47.792037 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 15:00:47.792188 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260928 15:00:47.793200 17483 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:00:47.793669 17483 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:00:47.801473 17483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Generated new TSK 0
I20260928 15:00:47.801642 17483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 15:00:47.831640 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:36033
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:47.915866 17492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:47.916149 17492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:47.916186 17492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:47.917328 17492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:47.917402 17492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:47.917447 17492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:00:47.918896 17492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:47.919170 17492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:47.919414 17492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:47.924655 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:47.924643 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:47.924780 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:47.924926 17492 server_base.cc:1034] running on GCE node
I20260928 15:00:47.925177 17492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:47.925419 17492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:47.926609 17492 hybrid_clock.cc:648] HybridClock initialized: now 1790607647926592 us; error 27 us; skew 500 ppm
Sep 28 15:00:47 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607647, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:47.929322 17492 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:00:47.929824 17492 webserver.cc:460] Webserver started at http://127.1.252.193:35281/ using document root <none> and password file <none>
I20260928 15:00:47.930073 17492 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:47.930145 17492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:47.930289 17492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:47.931341 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "82d4474c73c14cae86ef50fb12686a58"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "a3a03ffc6c4630b07c09ef561e606ee4"
server_key_iv: "00496eb8f294be6aab80f15049382acb"
server_key_version: "encryptionkey@0"
I20260928 15:00:47.931715 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "82d4474c73c14cae86ef50fb12686a58"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "a3a03ffc6c4630b07c09ef561e606ee4"
server_key_iv: "00496eb8f294be6aab80f15049382acb"
server_key_version: "encryptionkey@0"
I20260928 15:00:47.933127 17492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:00:47.933949 17507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:47.934181 17492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:00:47.934284 17492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "82d4474c73c14cae86ef50fb12686a58"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "a3a03ffc6c4630b07c09ef561e606ee4"
server_key_iv: "00496eb8f294be6aab80f15049382acb"
server_key_version: "encryptionkey@0"
I20260928 15:00:47.934371 17492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:47.946866 17492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:47.949576 17492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:47.949753 17492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:47.949995 17492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:47.950307 17492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:47.950367 17492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:47.950409 17492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:47.950438 17492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:47.956645 17492 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:36393
I20260928 15:00:47.956689 17621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:36393 every 8 connection(s)
I20260928 15:00:47.957011 17492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
Sep 28 15:00:47 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607647, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:47.966913 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:47.957764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:47493 (local address 127.1.252.254:36033)
0928 15:00:47.957851 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:47.957854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:47.958058 (+ 204us) server_negotiation.cc:407] Connection header received
0928 15:00:47.958557 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:47.958558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:47.958568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:47.958630 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:47.959551 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:47.960205 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:47.961184 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:47.961382 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:47.961445 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:47.963861 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:47.963862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:47.963862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:47.963870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:47.965432 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:47.965907 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:47.965908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:47.965908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:47.965963 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:47.966200 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:47.966201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:47.966201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:47.966312 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:47.966365 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:47.966758 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 15:00:47.966842 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:00:47.966971 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17492
I20260928 15:00:47.967068 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:00:47.967253 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:47.957727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:47493)
0928 15:00:47.957913 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:47.957956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:47.958466 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:47.958683 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:47.958694 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:47.959250 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:47.959453 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:47.959457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:47.960296 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:47.960297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:47.961106 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:47.961108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:47.961465 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:47.961466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:47.961565 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:47.962150 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:47.962158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:47.963751 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:47.965572 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:47.965580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:47.965581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:47.965828 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:47.966030 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:47.966031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:47.966031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:47.966135 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:47.966409 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:47.966417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:47.966549 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 15:00:47.966759 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 15:00:47.966967 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260928 15:00:47.967293 2035 external_mini_cluster.cc:1428] Setting key 898a15d6466c1a9a5623c57c344a44ce
I20260928 15:00:47.968004 17622 heartbeater.cc:347] Connected to a master server at 127.1.252.254:36033
I20260928 15:00:47.968145 17622 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:47.968356 17622 heartbeater.cc:511] Master 127.1.252.254:36033 requested a full tablet report, sending...
I20260928 15:00:47.968998 17427 ts_manager.cc:194] Registered new tserver with Master: 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393)
I20260928 15:00:47.969666 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:47493
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 15:00:47.996243 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:36033
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:48.078029 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:48.078284 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:48.078322 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:48.079493 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:48.079572 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:48.079622 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:00:48.081177 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:48.081460 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:48.081709 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:48.087796 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:48.087843 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:48.087828 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:48.088008 17627 server_base.cc:1034] running on GCE node
I20260928 15:00:48.088616 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:48.088881 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:48.090085 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790607648090041 us; error 63 us; skew 500 ppm
Sep 28 15:00:48 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607648, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:48.092911 17627 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:00:48.093425 17627 webserver.cc:460] Webserver started at http://127.1.252.194:44763/ using document root <none> and password file <none>
I20260928 15:00:48.093719 17627 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:48.093796 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:48.093952 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:48.095036 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "df4a4d7b39e44ea2bf509e56976b68c9"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "a14ca150085d3289d5191d5159c31fd2"
server_key_iv: "6f004b8a3960758449cef21c95a112b2"
server_key_version: "encryptionkey@0"
I20260928 15:00:48.095398 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "df4a4d7b39e44ea2bf509e56976b68c9"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "a14ca150085d3289d5191d5159c31fd2"
server_key_iv: "6f004b8a3960758449cef21c95a112b2"
server_key_version: "encryptionkey@0"
I20260928 15:00:48.096894 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:00:48.097852 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:48.098124 17627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:48.098246 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "df4a4d7b39e44ea2bf509e56976b68c9"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "a14ca150085d3289d5191d5159c31fd2"
server_key_iv: "6f004b8a3960758449cef21c95a112b2"
server_key_version: "encryptionkey@0"
I20260928 15:00:48.098335 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:48.118592 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:48.120744 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:48.120918 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:48.121172 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:48.121482 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:48.121551 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:48.121594 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:48.121622 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:48.127795 17627 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:36659
I20260928 15:00:48.127849 17756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:36659 every 8 connection(s)
I20260928 15:00:48.128168 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:00:48.131579 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17627
I20260928 15:00:48.131716 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:00:48.131939 2035 external_mini_cluster.cc:1428] Setting key 8b668b7a227718a3ff33377b73e935f8
Sep 28 15:00:48 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607648, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:48.139113 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.128945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:51797 (local address 127.1.252.254:36033)
0928 15:00:48.129026 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.129028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.129261 (+ 233us) server_negotiation.cc:407] Connection header received
0928 15:00:48.129779 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.129780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.129791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.129851 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:48.130910 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.131416 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.133130 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.133332 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.133391 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.136006 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:48.136007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:48.136008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:48.136015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:48.137626 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:48.138066 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:48.138067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:48.138068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:48.138131 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:48.138445 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:48.138446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:48.138447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:48.138564 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:48.138649 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.138943 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.139024 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:00:48.139319 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.128848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:51797)
0928 15:00:48.129103 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.129148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.129657 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.129951 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.129955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.130602 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:48.130822 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.130827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.131592 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.131593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.133037 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.133040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.133428 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.133429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.133616 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.134170 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:48.134182 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:48.135861 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:48.137736 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:48.137744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:48.137746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:48.137980 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:48.138216 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:48.138217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:48.138217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:48.138392 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:48.138647 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:48.138655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:48.138795 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.138905 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.139065 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260928 15:00:48.139868 17757 heartbeater.cc:347] Connected to a master server at 127.1.252.254:36033
I20260928 15:00:48.139998 17757 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:48.140220 17757 heartbeater.cc:511] Master 127.1.252.254:36033 requested a full tablet report, sending...
I20260928 15:00:48.140611 17427 ts_manager.cc:194] Registered new tserver with Master: df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
I20260928 15:00:48.141016 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:51797
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 15:00:48.161250 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:36033
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:00:48.244498 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:00:48.244707 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:00:48.244731 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:00:48.245914 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:00:48.245956 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:00:48.245982 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:00:48.247524 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:00:48.247789 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:00:48.248070 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:00:48.253293 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:48.253364 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:00:48.253505 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:00:48.253448 17762 server_base.cc:1034] running on GCE node
I20260928 15:00:48.253872 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:00:48.254128 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:00:48.255307 17762 hybrid_clock.cc:648] HybridClock initialized: now 1790607648255287 us; error 60 us; skew 500 ppm
Sep 28 15:00:48 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607648, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:00:48.258071 17762 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:00:48.258524 17762 webserver.cc:460] Webserver started at http://127.1.252.195:39663/ using document root <none> and password file <none>
I20260928 15:00:48.258764 17762 fs_manager.cc:359] Metadata directory not provided
I20260928 15:00:48.258836 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:00:48.258971 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:00:48.260227 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "158ac11269ca4836b585074888c55e9e"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "7e0c56f1e03513ad9ef94bcfa33c6afc"
server_key_iv: "862897f6121e1efa86ffae727dca2172"
server_key_version: "encryptionkey@0"
I20260928 15:00:48.260627 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "158ac11269ca4836b585074888c55e9e"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "7e0c56f1e03513ad9ef94bcfa33c6afc"
server_key_iv: "862897f6121e1efa86ffae727dca2172"
server_key_version: "encryptionkey@0"
I20260928 15:00:48.262038 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:48.262872 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:48.263130 17762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:00:48.263236 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "158ac11269ca4836b585074888c55e9e"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "7e0c56f1e03513ad9ef94bcfa33c6afc"
server_key_iv: "862897f6121e1efa86ffae727dca2172"
server_key_version: "encryptionkey@0"
I20260928 15:00:48.263322 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:00:48.277222 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:00:48.279639 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:00:48.279814 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:00:48.280117 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:00:48.280427 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:00:48.280493 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:48.280556 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:00:48.280583 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:00:48.287082 17762 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:36555
I20260928 15:00:48.287124 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:36555 every 8 connection(s)
I20260928 15:00:48.287447 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:00:48 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607648, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:00:48.296344 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17762
I20260928 15:00:48.296450 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:00:48.296643 2035 external_mini_cluster.cc:1428] Setting key 54267cdbca1f3987b4d361e5891640d6
I20260928 15:00:48.297483 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.288386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:40263 (local address 127.1.252.254:36033)
0928 15:00:48.288470 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.288472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.288652 (+ 180us) server_negotiation.cc:407] Connection header received
0928 15:00:48.289143 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.289144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.289155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.289208 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:48.290097 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.290603 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.291783 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.292039 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.292099 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.294309 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:48.294311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:48.294311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:48.294320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:48.295811 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:48.296298 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:48.296299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:48.296299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:48.296362 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:48.296664 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:48.296665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:48.296666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:48.296834 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:48.296907 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.297332 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.297421 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:00:48.297792 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.288333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:40263)
0928 15:00:48.288519 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.288557 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.289050 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.289262 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.289265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.289825 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:00:48.290020 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.290024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.290705 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.290706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.291686 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.291689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.292122 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.292123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.292227 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.292696 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:00:48.292704 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:00:48.294195 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:00:48.295923 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:48.295930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:48.295932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:48.296201 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:48.296439 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:00:48.296440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:00:48.296440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:00:48.296582 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:00:48.296971 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:00:48.296976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:00:48.297145 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.297277 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.297452 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260928 15:00:48.298342 17892 heartbeater.cc:347] Connected to a master server at 127.1.252.254:36033
I20260928 15:00:48.298435 17892 heartbeater.cc:464] Registering TS with master...
I20260928 15:00:48.298705 17892 heartbeater.cc:511] Master 127.1.252.254:36033 requested a full tablet report, sending...
I20260928 15:00:48.299150 17427 ts_manager.cc:194] Registered new tserver with Master: 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555)
I20260928 15:00:48.299486 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:40263
I20260928 15:00:48.308727 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:00:48.316413 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.310105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.1.252.254:36033)
0928 15:00:48.310187 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.310189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.310260 (+ 71us) server_negotiation.cc:407] Connection header received
0928 15:00:48.310354 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.310355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.310367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.310413 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:00:48.311012 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.311499 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.312378 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.312572 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.312652 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.313910 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:00:48.313911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:00:48.313912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:00:48.313918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:00:48.315119 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:48.315482 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:48.315483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:48.315483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:48.315583 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:00:48.315804 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:00:48.315805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:00:48.315806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:00:48.316015 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:00:48.316131 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.316232 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.316312 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:00:48.317943 17427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:00:48.319417 17427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:00:48.326063 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.321785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36393 (local address 127.0.0.1:55318)
0928 15:00:48.322246 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.322257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.322308 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.322579 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.322580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.322583 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:48.322669 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.322670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.324007 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.324008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.325011 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.325013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.325775 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.325856 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.325869 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.325939 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.325992 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260928 15:00:48.326094 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.321949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.1.252.193:36393)
0928 15:00:48.322179 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.322185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.322367 (+ 182us) server_negotiation.cc:407] Connection header received
0928 15:00:48.322408 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.322410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.322426 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.322495 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:48.322753 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.323866 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325112 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.325679 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.325834 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.325979 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.326018 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
I20260928 15:00:48.326236 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.321802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36659 (local address 127.0.0.1:59732)
0928 15:00:48.322269 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.322278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.322328 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.322574 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.322577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.322582 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:48.322658 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.322660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.323587 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.323588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.324908 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.324911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.325917 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.325979 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.325983 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.326031 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.326074 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260928 15:00:48.326275 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.322011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.1.252.194:36659)
0928 15:00:48.322188 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.322192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.322354 (+ 162us) server_negotiation.cc:407] Connection header received
0928 15:00:48.322395 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.322398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.322412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.322480 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:48.322751 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.323477 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.324999 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.325753 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325826 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.325880 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.326117 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.326165 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260928 15:00:48.326807 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.321784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:36555 (local address 127.0.0.1:58154)
0928 15:00:48.322032 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.322074 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.322141 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.322604 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.322604 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.322606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:48.322668 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.322669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.323996 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.323997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.325109 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.325111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.326048 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.326049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.326102 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.326105 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.326692 (+ 587us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.326739 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":45,"thread_start_us":45,"threads_started":1}
I20260928 15:00:48.326941 17555 tablet_service.cc:1467] Processing CreateTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee (DEFAULT_TABLE table=test-table [id=3ecaa750c1a94ba581686d46bb2e895f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:48.327009 17690 tablet_service.cc:1467] Processing CreateTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee (DEFAULT_TABLE table=test-table [id=3ecaa750c1a94ba581686d46bb2e895f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:48.327279 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.321938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.1.252.195:36555)
0928 15:00:48.322272 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.322275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.322283 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:00:48.322340 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.322342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.322361 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.322452 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:48.322755 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.323869 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325194 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.325893 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.325954 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.326004 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.327141 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.327188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":98,"threads_started":1}
I20260928 15:00:48.327315 17555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6333b35cb9ba4bc68be2a5c846e00fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:48.327390 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6333b35cb9ba4bc68be2a5c846e00fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:48.328541 17826 tablet_service.cc:1467] Processing CreateTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee (DEFAULT_TABLE table=test-table [id=3ecaa750c1a94ba581686d46bb2e895f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:00:48.328864 17826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6333b35cb9ba4bc68be2a5c846e00fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:00:48.330304 17909 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Bootstrap starting.
I20260928 15:00:48.330696 17910 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Bootstrap starting.
I20260928 15:00:48.331053 17909 tablet_bootstrap.cc:654] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:48.331331 17910 tablet_bootstrap.cc:654] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:48.331588 17909 log.cc:824] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:48.331946 17910 log.cc:824] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:48.332435 17909 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: No bootstrap required, opened a new log
I20260928 15:00:48.332520 17909 ts_tablet_manager.cc:1397] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:00:48.332711 17910 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: No bootstrap required, opened a new log
I20260928 15:00:48.332764 17910 ts_tablet_manager.cc:1397] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:00:48.334100 17909 raft_consensus.cc:348] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.334270 17909 raft_consensus.cc:374] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:48.334184 17910 raft_consensus.cc:348] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.334319 17909 raft_consensus.cc:729] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4d7b39e44ea2bf509e56976b68c9, State: Initialized, Role: FOLLOWER
I20260928 15:00:48.334343 17910 raft_consensus.cc:374] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:48.334383 17910 raft_consensus.cc:729] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d4474c73c14cae86ef50fb12686a58, State: Initialized, Role: FOLLOWER
I20260928 15:00:48.334438 17909 consensus_queue.cc:260] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.334540 17910 consensus_queue.cc:260] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.334748 17757 heartbeater.cc:503] Master 127.1.252.254:36033 was elected leader, sending a full tablet report...
I20260928 15:00:48.334846 17622 heartbeater.cc:503] Master 127.1.252.254:36033 was elected leader, sending a full tablet report...
I20260928 15:00:48.334852 17910 ts_tablet_manager.cc:1428] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:00:48.334746 17909 ts_tablet_manager.cc:1428] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:00:48.335207 17912 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Bootstrap starting.
I20260928 15:00:48.335832 17912 tablet_bootstrap.cc:654] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Neither blocks nor log segments found. Creating new log.
I20260928 15:00:48.336331 17912 log.cc:824] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:00:48.337024 17912 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: No bootstrap required, opened a new log
I20260928 15:00:48.337078 17912 ts_tablet_manager.cc:1397] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:00:48.338595 17912 raft_consensus.cc:348] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.338729 17912 raft_consensus.cc:374] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:00:48.338752 17912 raft_consensus.cc:729] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158ac11269ca4836b585074888c55e9e, State: Initialized, Role: FOLLOWER
I20260928 15:00:48.338860 17912 consensus_queue.cc:260] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.339128 17912 ts_tablet_manager.cc:1428] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:00:48.339164 17892 heartbeater.cc:503] Master 127.1.252.254:36033 was elected leader, sending a full tablet report...
I20260928 15:00:48.358750 17914 raft_consensus.cc:482] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:00:48.358853 17914 raft_consensus.cc:504] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.359361 17914 leader_election.cc:288] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393)
I20260928 15:00:48.363359 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.359622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:46761 (local address 127.1.252.193:36393)
0928 15:00:48.359684 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.359686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.359772 (+ 86us) server_negotiation.cc:407] Connection header received
0928 15:00:48.359800 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.359801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.359810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.359863 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:48.360057 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.360822 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.362344 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.362919 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.362999 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.363006 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.363246 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.363289 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:00:48.363359 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.359550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:34255 (local address 127.1.252.195:36555)
0928 15:00:48.359609 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.359611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.359621 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:00:48.359736 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.359760 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.359770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.359854 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:00:48.360057 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.360801 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.362344 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.362919 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.362998 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.363006 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.363247 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.363288 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:00:48.363366 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.359504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:36555 (local address 127.1.252.194:34255)
0928 15:00:48.359560 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.359573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.359617 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.359860 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.359861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.359862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:48.359968 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.359970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.360910 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.360911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.362239 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.362242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.363037 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.363038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.363150 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.363152 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.363219 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.363270 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":78}
I20260928 15:00:48.363373 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.359548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36393 (local address 127.1.252.194:46761)
0928 15:00:48.359693 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.359702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:00:48.359754 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:00:48.359876 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:00:48.359876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:00:48.359877 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:00:48.359969 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.359970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.360910 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.360911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.362239 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:00:48.362242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.363037 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.363038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:00:48.363146 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.363151 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:00:48.363219 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:00:48.363270 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":95,"thread_start_us":101,"threads_started":1}
I20260928 15:00:48.363797 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6333b35cb9ba4bc68be2a5c846e00fee" candidate_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d4474c73c14cae86ef50fb12686a58" is_pre_election: true
I20260928 15:00:48.363801 17846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6333b35cb9ba4bc68be2a5c846e00fee" candidate_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158ac11269ca4836b585074888c55e9e" is_pre_election: true
I20260928 15:00:48.363979 17846 raft_consensus.cc:2454] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4a4d7b39e44ea2bf509e56976b68c9 in term 0.
I20260928 15:00:48.363979 17576 raft_consensus.cc:2454] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4a4d7b39e44ea2bf509e56976b68c9 in term 0.
I20260928 15:00:48.364204 17646 leader_election.cc:302] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58, df4a4d7b39e44ea2bf509e56976b68c9; no voters:
I20260928 15:00:48.364351 17914 raft_consensus.cc:2781] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:00:48.364404 17914 raft_consensus.cc:482] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:00:48.364429 17914 raft_consensus.cc:3037] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:48.365141 17914 raft_consensus.cc:504] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.365293 17914 leader_election.cc:288] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [CANDIDATE]: Term 1 election: Requested vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393)
I20260928 15:00:48.365460 17846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6333b35cb9ba4bc68be2a5c846e00fee" candidate_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158ac11269ca4836b585074888c55e9e"
I20260928 15:00:48.365466 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6333b35cb9ba4bc68be2a5c846e00fee" candidate_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d4474c73c14cae86ef50fb12686a58"
I20260928 15:00:48.365546 17846 raft_consensus.cc:3037] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:48.365542 17576 raft_consensus.cc:3037] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:00:48.366449 17846 raft_consensus.cc:2454] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4a4d7b39e44ea2bf509e56976b68c9 in term 1.
I20260928 15:00:48.366449 17576 raft_consensus.cc:2454] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4a4d7b39e44ea2bf509e56976b68c9 in term 1.
I20260928 15:00:48.366597 17646 leader_election.cc:302] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58, df4a4d7b39e44ea2bf509e56976b68c9; no voters:
I20260928 15:00:48.366786 17914 raft_consensus.cc:2781] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:00:48.366981 17914 raft_consensus.cc:686] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 LEADER]: Becoming Leader. State: Replica: df4a4d7b39e44ea2bf509e56976b68c9, State: Running, Role: LEADER
I20260928 15:00:48.367102 17914 consensus_queue.cc:237] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:00:48.367959 17427 catalog_manager.cc:5495] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } health_report { overall_health: HEALTHY } } }
W20260928 15:00:48.378805 17758 tablet.cc:2369] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:00:48.383764 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.381115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.1.252.194:36659)
0928 15:00:48.381167 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:00:48.381169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:00:48.381179 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:00:48.381216 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:00:48.381217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:00:48.381239 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:00:48.381328 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:00:48.381469 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.381958 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.382803 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:00:48.383008 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:00:48.383109 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:00:48.383288 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:00:48.383494 (+ 206us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:00:48.383606 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:00:48.383648 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 15:00:48.383676 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:00:48.385730 17576 raft_consensus.cc:1261] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Refusing update from remote peer df4a4d7b39e44ea2bf509e56976b68c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:48.385748 17846 raft_consensus.cc:1261] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Refusing update from remote peer df4a4d7b39e44ea2bf509e56976b68c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:00:48.386178 17919 consensus_queue.cc:1035] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:48.386327 17914 consensus_queue.cc:1035] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:00:48.388866 17922 mvcc.cc:205] Tried to move back new op lower bound from 7334328927784947712 to 7334328927712223232. Current Snapshot: MvccSnapshot[applied={T|T < 7334328927784947712}]
I20260928 15:00:48.389786 17924 mvcc.cc:205] Tried to move back new op lower bound from 7334328927784947712 to 7334328927712223232. Current Snapshot: MvccSnapshot[applied={T|T < 7334328927784947712}]
I20260928 15:00:48.391731 17428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:48.391834 17428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:00:48.392748 17428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Sending DeleteTablet for 3 replicas of tablet 6333b35cb9ba4bc68be2a5c846e00fee
I20260928 15:00:48.393131 17826 tablet_service.cc:1514] Processing DeleteTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:58154
I20260928 15:00:48.393177 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17492
I20260928 15:00:48.393178 17555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:55318
I20260928 15:00:48.393739 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59732
I20260928 15:00:48.394104 17932 tablet_replica.cc:330] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: stopping tablet replica
I20260928 15:00:48.394225 17932 raft_consensus.cc:2229] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:00:48.394353 17932 raft_consensus.cc:2258] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:48.394601 17931 tablet_replica.cc:330] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: stopping tablet replica
I20260928 15:00:48.394698 17931 raft_consensus.cc:2229] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:00:48.394801 17931 raft_consensus.cc:2258] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:00:48.395112 17932 ts_tablet_manager.cc:1905] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:48.396366 17931 ts_tablet_manager.cc:1905] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:00:48.397221 17932 ts_tablet_manager.cc:1918] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:48.397303 17932 log.cc:1196] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/6333b35cb9ba4bc68be2a5c846e00fee
I20260928 15:00:48.397571 17932 ts_tablet_manager.cc:1939] T 6333b35cb9ba4bc68be2a5c846e00fee P df4a4d7b39e44ea2bf509e56976b68c9: Deleting consensus metadata
I20260928 15:00:48.398039 17412 catalog_manager.cc:4841] TS df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): tablet 6333b35cb9ba4bc68be2a5c846e00fee (table test-table [id=3ecaa750c1a94ba581686d46bb2e895f]) successfully deleted
I20260928 15:00:48.398310 17931 ts_tablet_manager.cc:1918] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:00:48.398435 17931 log.cc:1196] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/6333b35cb9ba4bc68be2a5c846e00fee
I20260928 15:00:48.398784 17931 ts_tablet_manager.cc:1939] T 6333b35cb9ba4bc68be2a5c846e00fee P 158ac11269ca4836b585074888c55e9e: Deleting consensus metadata
I20260928 15:00:48.399319 17414 catalog_manager.cc:4841] TS 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): tablet 6333b35cb9ba4bc68be2a5c846e00fee (table test-table [id=3ecaa750c1a94ba581686d46bb2e895f]) successfully deleted
W20260928 15:00:48.402302 17415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:36393 (error 108)
I20260928 15:00:48.402558 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17627
I20260928 15:00:48.403216 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:00:48.402716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36393 (local address 127.0.0.1:55334)
0928 15:00:48.403127 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 15:00:48.403175 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:36393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
W20260928 15:00:48.403393 17415 catalog_manager.cc:4584] TS 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6333b35cb9ba4bc68be2a5c846e00fee: Network error: Client connection negotiation failed: client connection to 127.1.252.193:36393: connect: Connection refused (error 111)
I20260928 15:00:48.408100 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17762
I20260928 15:00:48.412915 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17396
I20260928 15:01:08.418431 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:36033
--webserver_interface=127.1.252.254
--webserver_port=37727
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:08.506421 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:08.506644 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:08.506666 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:08.507812 17934 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:01:08.507851 17934 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:01:08.507865 17934 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:01:08.507877 17934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:08.507957 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:08.507977 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:08.508001 17934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:08.508019 17934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:08.509573 17934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:08.509825 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:08.510067 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:08.519189 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:08.519218 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:08.519256 17934 server_base.cc:1034] running on GCE node
W20260928 15:01:08.519210 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:08.519762 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:08.520148 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:08.521292 17934 hybrid_clock.cc:648] HybridClock initialized: now 1790607668521264 us; error 41 us; skew 500 ppm
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:08.523957 17934 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:01:08.524403 17934 webserver.cc:460] Webserver started at http://127.1.252.254:37727/ using document root <none> and password file <none>
I20260928 15:01:08.524642 17934 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:08.524721 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:08.526068 17934 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:08.526835 17950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.527040 17934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:08.527146 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "7c41c0c8f9604f75862b9a0ed05017d8"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "41e35cb3f26e30be94235990c77c8e3d"
server_key_iv: "efbf196ff05acf5a234c9f0df313522c"
server_key_version: "encryptionkey@0"
I20260928 15:01:08.527547 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:08.540743 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:08.543032 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:08.543195 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:08.547366 17934 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:36033
I20260928 15:01:08.547402 18003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:36033 every 8 connection(s)
I20260928 15:01:08.547817 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:08.549302 18004 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:01:08.549891 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap starting.
I20260928 15:01:08.553951 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 17934
I20260928 15:01:08.554246 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:36393
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:36033
--webserver_port=35281
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:01:08.558902 18004 log.cc:824] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:08.560889 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:01:08.561200 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap complete.
I20260928 15:01:08.563481 18004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:08.563758 18004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c41c0c8f9604f75862b9a0ed05017d8, State: Initialized, Role: FOLLOWER
I20260928 15:01:08.563979 18004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:08.564074 18004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:08.564131 18004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:08.564178 18004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:01:08.565140 18004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:08.565258 18004 leader_election.cc:302] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c41c0c8f9604f75862b9a0ed05017d8; no voters:
I20260928 15:01:08.565444 18004 leader_election.cc:288] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:01:08.565479 18008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:01:08.565593 18008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 LEADER]: Becoming Leader. State: Replica: 7c41c0c8f9604f75862b9a0ed05017d8, State: Running, Role: LEADER
I20260928 15:01:08.565680 18008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:08.565714 18004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:08.565927 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } } }
I20260928 15:01:08.566044 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:08.565923 18008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c41c0c8f9604f75862b9a0ed05017d8. Latest consensus state: current_term: 2 leader_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } } }
I20260928 15:01:08.566237 18008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:08.566460 18012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:08.567250 18012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:08.567389 18012 catalog_manager.cc:1259] Loaded cluster ID: ff7dff5a5fd04852be0a91d97695c07d
I20260928 15:01:08.567700 18012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:08.568948 18012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:08.569121 18012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Loaded TSK: 0
I20260928 15:01:08.582023 18012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Generated new TSK 1
I20260928 15:01:08.582118 18012 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:01:08.642431 18006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:08.642621 18006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:08.642643 18006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:08.643819 18006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:08.643867 18006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:08.643913 18006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:08.645385 18006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:36393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=35281
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:08.645606 18006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:08.645854 18006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:08.652019 18032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:08.652241 18006 server_base.cc:1034] running on GCE node
W20260928 15:01:08.652010 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:08.651988 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:08.652638 18006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:08.652937 18006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:08.654138 18006 hybrid_clock.cc:648] HybridClock initialized: now 1790607668654124 us; error 35 us; skew 500 ppm
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:08.656976 18006 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:01:08.657510 18006 webserver.cc:460] Webserver started at http://127.1.252.193:35281/ using document root <none> and password file <none>
I20260928 15:01:08.657766 18006 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:08.657829 18006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:08.659475 18006 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:01:08.660283 18041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.660452 18006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:08.660554 18006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "82d4474c73c14cae86ef50fb12686a58"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "a3a03ffc6c4630b07c09ef561e606ee4"
server_key_iv: "00496eb8f294be6aab80f15049382acb"
server_key_version: "encryptionkey@0"
I20260928 15:01:08.660930 18006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:08.676553 18006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:08.679122 18006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:08.679278 18006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:08.679605 18006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:08.680222 18048 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:01:08.681836 18006 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:01:08.681892 18006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:01:08.681931 18006 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:01:08.682673 18006 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:01:08.682720 18006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:08.682767 18048 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Bootstrap starting.
I20260928 15:01:08.691128 18006 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:36393
I20260928 15:01:08.691285 18155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:36393 every 8 connection(s)
I20260928 15:01:08.691660 18006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:08.695740 18048 log.cc:824] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:08.698325 18048 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:01:08.698751 18048 tablet_bootstrap.cc:492] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Bootstrap complete.
I20260928 15:01:08.699335 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18006
I20260928 15:01:08.699707 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:36659
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:36033
--webserver_port=44763
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:01:08.700013 18048 ts_tablet_manager.cc:1397] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.011s
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:08.701670 18048 raft_consensus.cc:348] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:01:08.702153 18048 raft_consensus.cc:729] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d4474c73c14cae86ef50fb12686a58, State: Initialized, Role: FOLLOWER
I20260928 15:01:08.702345 18048 consensus_queue.cc:260] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } }
I20260928 15:01:08.702656 18048 ts_tablet_manager.cc:1428] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 15:01:08.705149 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.692387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:39427)
0928 15:01:08.692662 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.692702 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:08.693259 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:08.693823 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:08.693827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:08.694155 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:08.694410 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.694416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.696337 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.696338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.697126 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.697128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.698074 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.698076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.698204 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.698869 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:08.698877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:08.700812 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:08.703260 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:08.703266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:08.703269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:08.703512 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:08.703816 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:08.703817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:08.703818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:08.704012 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:08.704434 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:08.704441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:08.704590 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 15:01:08.704713 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 15:01:08.704903 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260928 15:01:08.705142 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.692500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:39427 (local address 127.1.252.254:36033)
0928 15:01:08.692748 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.692759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.692798 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:01:08.693502 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.693511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.693590 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.693780 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:08.694579 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.696179 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.697221 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.697901 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.698029 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.700940 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:08.700944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:08.700945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:08.700958 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:08.703150 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.703628 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.703631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.703631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.703732 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.704108 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.704109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.704110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.704350 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:08.704454 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:08.704768 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 15:01:08.704903 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
I20260928 15:01:08.705823 18156 heartbeater.cc:347] Connected to a master server at 127.1.252.254:36033
I20260928 15:01:08.705907 18156 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:08.706111 18156 heartbeater.cc:511] Master 127.1.252.254:36033 requested a full tablet report, sending...
I20260928 15:01:08.706907 17967 ts_manager.cc:194] Registered new tserver with Master: 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393)
I20260928 15:01:08.708105 17967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:39427
I20260928 15:01:08.709105 18156 heartbeater.cc:503] Master 127.1.252.254:36033 was elected leader, sending a full tablet report...
I20260928 15:01:08.715790 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.707599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.1.252.193:36393)
0928 15:01:08.712069 (+ 4470us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.712073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.712087 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:01:08.712259 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.712269 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.712283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.712357 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:08.712512 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.713160 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.714657 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.715354 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.715413 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.715452 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:08.715699 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:01:08.715737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4435,"thread_start_us":90,"threads_started":1}
I20260928 15:01:08.715808 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.707511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36393 (local address 127.0.0.1:48170)
0928 15:01:08.712002 (+ 4491us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.712042 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:08.712107 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:08.712375 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:08.712378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:08.712380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:08.712444 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.712446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.713232 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.713233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.714550 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.714561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.715440 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.715442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.715559 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.715577 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 15:01:08.715664 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:01:08.715726 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4456,"thread_start_us":95,"threads_started":1}
I20260928 15:01:08.716424 18090 tablet_service.cc:1514] Processing DeleteTablet for tablet 6333b35cb9ba4bc68be2a5c846e00fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:00:48 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:48170
I20260928 15:01:08.716928 18165 tablet_replica.cc:330] stopping tablet replica
I20260928 15:01:08.717032 18165 raft_consensus.cc:2229] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:08.717099 18165 raft_consensus.cc:2258] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:08.717403 18165 ts_tablet_manager.cc:1905] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:08.719825 18165 ts_tablet_manager.cc:1918] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:08.719928 18165 log.cc:1196] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal/wals/6333b35cb9ba4bc68be2a5c846e00fee
I20260928 15:01:08.720161 18165 ts_tablet_manager.cc:1939] T 6333b35cb9ba4bc68be2a5c846e00fee P 82d4474c73c14cae86ef50fb12686a58: Deleting consensus metadata
I20260928 15:01:08.720815 17954 catalog_manager.cc:4841] TS 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393): tablet 6333b35cb9ba4bc68be2a5c846e00fee (table test-table [id=3ecaa750c1a94ba581686d46bb2e895f]) successfully deleted
W20260928 15:01:08.784161 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:08.784397 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:08.784435 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:08.785552 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:08.785619 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:08.785671 18161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:08.787107 18161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:36659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=44763
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:08.787359 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:08.787585 18161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:08.792620 18171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:08.792716 18173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:08.792644 18170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:08.792831 18161 server_base.cc:1034] running on GCE node
I20260928 15:01:08.793058 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:08.793327 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:08.794489 18161 hybrid_clock.cc:648] HybridClock initialized: now 1790607668794483 us; error 38 us; skew 500 ppm
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:08.797405 18161 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:01:08.797850 18161 webserver.cc:460] Webserver started at http://127.1.252.194:44763/ using document root <none> and password file <none>
I20260928 15:01:08.798090 18161 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:08.798154 18161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:08.799623 18161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:01:08.800335 18181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.800522 18161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:08.800643 18161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "df4a4d7b39e44ea2bf509e56976b68c9"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "a14ca150085d3289d5191d5159c31fd2"
server_key_iv: "6f004b8a3960758449cef21c95a112b2"
server_key_version: "encryptionkey@0"
I20260928 15:01:08.801033 18161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:08.810382 18161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:08.812603 18161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:08.812762 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:08.812981 18161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:08.813297 18161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:08.813351 18161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.813391 18161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:08.813418 18161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.819501 18161 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:36659
I20260928 15:01:08.819540 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:36659 every 8 connection(s)
I20260928 15:01:08.819965 18161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:08.825237 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18161
I20260928 15:01:08.825601 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:36555
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:36033
--webserver_port=39663
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:08.831478 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.820812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:45663 (local address 127.1.252.254:36033)
0928 15:01:08.820901 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.820904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.821110 (+ 206us) server_negotiation.cc:407] Connection header received
0928 15:01:08.821740 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.821746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.821764 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.821819 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:08.822873 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.823434 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.824624 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.824858 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.824956 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.827718 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:08.827721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:08.827721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:08.827733 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:08.829637 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.830121 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.830123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.830123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.830195 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.830609 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.830610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.830610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.830789 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:08.830877 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:08.831210 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 15:01:08.831323 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:01:08.831489 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.820735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:45663)
0928 15:01:08.820966 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.821016 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:08.821605 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:08.821890 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:08.821894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:08.822552 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:08.822764 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.822769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.823548 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.823549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.824526 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.824529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.824967 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.824968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.825155 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.825974 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:08.825983 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:08.827607 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:08.829725 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:08.829732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:08.829733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:08.830036 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:08.830299 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:08.830300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:08.830301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:08.830490 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:08.830872 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:08.830877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:08.831030 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 15:01:08.831160 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 15:01:08.831305 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260928 15:01:08.832258 18296 heartbeater.cc:347] Connected to a master server at 127.1.252.254:36033
I20260928 15:01:08.832383 18296 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:08.832624 18296 heartbeater.cc:511] Master 127.1.252.254:36033 requested a full tablet report, sending...
I20260928 15:01:08.833103 17967 ts_manager.cc:194] Registered new tserver with Master: df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
I20260928 15:01:08.833515 17967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:45663
W20260928 15:01:08.911772 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:08.912092 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:08.912134 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:08.913295 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:08.913360 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:08.913407 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:08.914884 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:36555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=39663
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:08.915154 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:08.915387 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:08.921025 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:08.921010 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:08.921005 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:08.921280 18299 server_base.cc:1034] running on GCE node
I20260928 15:01:08.921521 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:08.921772 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:08.922915 18299 hybrid_clock.cc:648] HybridClock initialized: now 1790607668922902 us; error 28 us; skew 500 ppm
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:08.925449 18299 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:01:08.925894 18299 webserver.cc:460] Webserver started at http://127.1.252.195:39663/ using document root <none> and password file <none>
I20260928 15:01:08.926141 18299 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:08.926231 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:08.927691 18299 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:08.928362 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.928658 18299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:08.928769 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "158ac11269ca4836b585074888c55e9e"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "7e0c56f1e03513ad9ef94bcfa33c6afc"
server_key_iv: "862897f6121e1efa86ffae727dca2172"
server_key_version: "encryptionkey@0"
I20260928 15:01:08.929129 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:08.957649 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:08.959923 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:08.960098 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:08.960345 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:08.960659 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:08.960718 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.960762 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:08.960810 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:08.967777 18299 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:36555
I20260928 15:01:08.967823 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:36555 every 8 connection(s)
I20260928 15:01:08.968227 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:08.971936 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18299
Sep 28 15:01:08 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607668, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:08.978575 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.969009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:51097 (local address 127.1.252.254:36033)
0928 15:01:08.969057 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.969059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.969263 (+ 204us) server_negotiation.cc:407] Connection header received
0928 15:01:08.969757 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.969758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.969770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.969854 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:08.970839 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.971347 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.972627 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.972950 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.973029 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.975286 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:08.975288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:08.975289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:08.975297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:08.977070 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.977579 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.977580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.977581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.977638 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.977906 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.977907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.977908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.978033 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:08.978107 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:08.978389 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 15:01:08.978477 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:01:08.978785 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.968996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:51097)
0928 15:01:08.969133 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.969171 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:08.969655 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:08.969909 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:08.969912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:08.970502 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:08.970722 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.970727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.971441 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.971442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.972548 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.972551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.973038 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.973038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:08.973144 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.973715 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:08.973724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:08.975187 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:08.977167 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:08.977174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:08.977176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:08.977501 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:08.977715 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:08.977716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:08.977716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:08.977835 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:08.978113 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:08.978120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:08.978259 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 15:01:08.978367 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:01:08.978538 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260928 15:01:08.979323 18429 heartbeater.cc:347] Connected to a master server at 127.1.252.254:36033
I20260928 15:01:08.979446 18429 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:08.979487 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.972394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.1.252.254:36033)
0928 15:01:08.972653 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.972654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.972661 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:01:08.972744 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.972745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.972754 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.972799 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:08.973466 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.974016 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.974886 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.975106 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.975181 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.976386 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:08.976387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:08.976387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:08.976393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:08.978016 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.978429 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.978430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.978431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.978500 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:08.978734 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:08.978735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:08.978736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:08.978873 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:08.979077 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:08.979288 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 15:01:08.979369 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":69,"threads_started":1}
I20260928 15:01:08.979712 18429 heartbeater.cc:511] Master 127.1.252.254:36033 requested a full tablet report, sending...
I20260928 15:01:08.980192 17966 ts_manager.cc:194] Registered new tserver with Master: 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555)
I20260928 15:01:08.980602 17966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:51097
I20260928 15:01:08.991384 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:08.994560 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.991843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.1.252.254:36033)
0928 15:01:08.991938 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.991940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.992036 (+ 96us) server_negotiation.cc:407] Connection header received
0928 15:01:08.992104 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.992104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.992136 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.992200 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:08.992384 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.992885 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.993777 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:08.993979 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:08.994036 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:08.994242 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:08.994347 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:08.994421 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:08.994462 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 15:01:08.994490 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 15:01:08.994901 17966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:08.996693 17966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:08.999397 18090 tablet_service.cc:1467] Processing CreateTablet for tablet 98c1d790e675420f85c95d6592151a9e (DEFAULT_TABLE table=test-table [id=f0435bf43e1449158a44783a20e3a7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:08.999671 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c1d790e675420f85c95d6592151a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:09.001178 18048 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Bootstrap starting.
I20260928 15:01:09.001510 18048 tablet_bootstrap.cc:654] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:09.002027 18048 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: No bootstrap required, opened a new log
I20260928 15:01:09.002094 18048 ts_tablet_manager.cc:1397] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:09.002215 18048 raft_consensus.cc:348] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.002287 18048 raft_consensus.cc:374] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:09.002344 18048 raft_consensus.cc:729] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d4474c73c14cae86ef50fb12686a58, State: Initialized, Role: FOLLOWER
I20260928 15:01:09.002390 18048 consensus_queue.cc:260] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.002473 18048 ts_tablet_manager.cc:1428] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:09.002872 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.998844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:36555 (local address 127.0.0.1:34356)
0928 15:01:08.998892 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.998903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:08.998940 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:08.999359 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:08.999360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:08.999361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:08.999412 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:08.999413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.000498 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.000500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.001563 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.001565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.002639 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.002640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.002703 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.002707 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:09.002760 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:01:09.002807 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 15:01:09.003111 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.998931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.1.252.195:36555)
0928 15:01:08.999177 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:08.999182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:08.999193 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:01:08.999233 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:08.999235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:08.999252 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:08.999348 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:08.999474 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.000398 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.001676 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.002489 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.002616 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.002683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.002959 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.003010 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":160,"threads_started":1}
I20260928 15:01:09.003705 18362 tablet_service.cc:1467] Processing CreateTablet for tablet 98c1d790e675420f85c95d6592151a9e (DEFAULT_TABLE table=test-table [id=f0435bf43e1449158a44783a20e3a7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:09.004061 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c1d790e675420f85c95d6592151a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:09.005291 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.998917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36659 (local address 127.0.0.1:43730)
0928 15:01:08.999117 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:08.999127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:08.999199 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:09.001448 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:09.001449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:09.001451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:09.001487 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.001488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.003009 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.003010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.004299 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.004301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.005041 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.005042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.005104 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.005108 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:09.005168 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 15:01:09.005220 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":153,"threads_started":1}
I20260928 15:01:09.005308 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:08.999269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.1.252.194:36659)
0928 15:01:09.001205 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.001210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.001221 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:01:09.001260 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.001262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.001278 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.001356 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:09.001817 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.002899 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.004394 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.004951 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.005040 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.005104 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.005182 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.005223 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1895,"thread_start_us":94,"threads_started":1}
I20260928 15:01:09.006036 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 98c1d790e675420f85c95d6592151a9e (DEFAULT_TABLE table=test-table [id=f0435bf43e1449158a44783a20e3a7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:09.006358 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c1d790e675420f85c95d6592151a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:09.007388 18439 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Bootstrap starting.
I20260928 15:01:09.008028 18439 tablet_bootstrap.cc:654] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:09.008548 18439 log.cc:824] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:09.009305 18439 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: No bootstrap required, opened a new log
I20260928 15:01:09.009383 18439 ts_tablet_manager.cc:1397] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:01:09.009418 18441 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Bootstrap starting.
I20260928 15:01:09.009929 18441 tablet_bootstrap.cc:654] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:09.010460 18441 log.cc:824] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:09.011139 18441 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: No bootstrap required, opened a new log
I20260928 15:01:09.011190 18441 ts_tablet_manager.cc:1397] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:09.011058 18439 raft_consensus.cc:348] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.011227 18439 raft_consensus.cc:374] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:09.011268 18439 raft_consensus.cc:729] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158ac11269ca4836b585074888c55e9e, State: Initialized, Role: FOLLOWER
I20260928 15:01:09.011389 18439 consensus_queue.cc:260] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.011679 18439 ts_tablet_manager.cc:1428] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 15:01:09.011726 18429 heartbeater.cc:503] Master 127.1.252.254:36033 was elected leader, sending a full tablet report...
I20260928 15:01:09.012919 18441 raft_consensus.cc:348] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.013051 18441 raft_consensus.cc:374] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:09.013072 18441 raft_consensus.cc:729] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4d7b39e44ea2bf509e56976b68c9, State: Initialized, Role: FOLLOWER
I20260928 15:01:09.013140 18441 consensus_queue.cc:260] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.013337 18441 ts_tablet_manager.cc:1428] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:09.013391 18296 heartbeater.cc:503] Master 127.1.252.254:36033 was elected leader, sending a full tablet report...
W20260928 15:01:09.070582 18297 tablet.cc:2369] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:09.111032 18443 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:09.111161 18443 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.111503 18443 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659), 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393)
I20260928 15:01:09.114989 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.111756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36393 (local address 127.1.252.195:60499)
0928 15:01:09.111830 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:09.111846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:09.111942 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:09.112144 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:09.112145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:09.112147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:09.112235 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.112237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.112982 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.112983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.113860 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.113862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.114677 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.114678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.114781 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.114785 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:09.114851 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:01:09.114908 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":22}
I20260928 15:01:09.114988 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.111930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:60499 (local address 127.1.252.193:36393)
0928 15:01:09.111998 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.112000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.112018 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:01:09.112052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.112053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.112065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.112123 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:09.112326 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.112894 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.113949 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.114604 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.114670 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.114679 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.114857 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.114894 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:01:09.115056 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.111839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:59879 (local address 127.1.252.194:36659)
0928 15:01:09.111903 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.111905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.112155 (+ 250us) server_negotiation.cc:407] Connection header received
0928 15:01:09.112185 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.112186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.112196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.112254 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:09.112540 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.113081 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.114150 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.114731 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.114789 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.114795 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.114955 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.114989 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:01:09.115093 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.111758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36659 (local address 127.1.252.195:59879)
0928 15:01:09.112040 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:09.112050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:09.112112 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:09.112320 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:09.112321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:09.112324 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:09.112401 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.112403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.113166 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.113167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.114033 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:09.114036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.114809 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.114810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:09.114878 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.114881 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:09.114934 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:01:09.114973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":71,"thread_start_us":130,"threads_started":1}
I20260928 15:01:09.115334 18110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c1d790e675420f85c95d6592151a9e" candidate_uuid: "158ac11269ca4836b585074888c55e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d4474c73c14cae86ef50fb12686a58" is_pre_election: true
I20260928 15:01:09.115418 18249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c1d790e675420f85c95d6592151a9e" candidate_uuid: "158ac11269ca4836b585074888c55e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" is_pre_election: true
I20260928 15:01:09.115494 18110 raft_consensus.cc:2454] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ac11269ca4836b585074888c55e9e in term 0.
I20260928 15:01:09.115628 18249 raft_consensus.cc:2454] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158ac11269ca4836b585074888c55e9e in term 0.
I20260928 15:01:09.115728 18318 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158ac11269ca4836b585074888c55e9e, 82d4474c73c14cae86ef50fb12686a58; no voters:
I20260928 15:01:09.115876 18443 raft_consensus.cc:2781] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:09.115965 18443 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:09.116003 18443 raft_consensus.cc:3037] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:09.116712 18443 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.117002 18443 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [CANDIDATE]: Term 1 election: Requested vote from peers df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659), 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393)
I20260928 15:01:09.117025 18249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c1d790e675420f85c95d6592151a9e" candidate_uuid: "158ac11269ca4836b585074888c55e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4d7b39e44ea2bf509e56976b68c9"
I20260928 15:01:09.117089 18249 raft_consensus.cc:3037] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:09.117187 18110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c1d790e675420f85c95d6592151a9e" candidate_uuid: "158ac11269ca4836b585074888c55e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d4474c73c14cae86ef50fb12686a58"
I20260928 15:01:09.117254 18110 raft_consensus.cc:3037] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:09.117754 18249 raft_consensus.cc:2454] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ac11269ca4836b585074888c55e9e in term 1.
I20260928 15:01:09.117939 18315 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158ac11269ca4836b585074888c55e9e, df4a4d7b39e44ea2bf509e56976b68c9; no voters:
I20260928 15:01:09.118021 18443 raft_consensus.cc:2781] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:09.118155 18443 raft_consensus.cc:686] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 1 LEADER]: Becoming Leader. State: Replica: 158ac11269ca4836b585074888c55e9e, State: Running, Role: LEADER
I20260928 15:01:09.118216 18443 consensus_queue.cc:237] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:09.118422 18110 raft_consensus.cc:2454] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158ac11269ca4836b585074888c55e9e in term 1.
I20260928 15:01:09.119009 17966 catalog_manager.cc:5495] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 158ac11269ca4836b585074888c55e9e (127.1.252.195). New cstate: current_term: 1 leader_uuid: "158ac11269ca4836b585074888c55e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } health_report { overall_health: UNKNOWN } } }
I20260928 15:01:09.148083 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.145930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.1.252.195:36555)
0928 15:01:09.146003 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.146005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.146012 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:01:09.146045 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.146045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.146062 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.146127 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:09.146226 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.146715 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.147425 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.147634 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.147684 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.147827 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:09.147870 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 15:01:09.148008 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34370: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 15:01:09.148177 18435 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34370: token expired
I20260928 15:01:09.154474 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.148558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.1.252.254:36033)
0928 15:01:09.148637 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.148639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.148648 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:01:09.148675 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.148676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.148683 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.148748 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:09.149256 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.149719 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.150412 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.150610 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.150679 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.151878 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:09.151880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:09.151880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:09.151906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:09.153206 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.153671 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.153673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.153673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.153733 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:09.153933 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:09.153934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:09.153935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:09.154077 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:09.154186 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.154287 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.154371 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:01:09.157513 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.155388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.1.252.195:36555)
0928 15:01:09.155445 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:09.155447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:09.155455 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:01:09.155477 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:09.155483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:09.155492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:09.155533 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:09.155636 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.156197 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.156901 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:09.157081 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:09.157128 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:09.157236 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:09.157311 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:09.157350 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:09.157447 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 15:01:09.157464 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 15:01:09.159422 18249 raft_consensus.cc:1261] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Refusing update from remote peer 158ac11269ca4836b585074888c55e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:09.159425 18110 raft_consensus.cc:1261] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Refusing update from remote peer 158ac11269ca4836b585074888c55e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:09.159795 18446 consensus_queue.cc:1035] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:09.159936 18443 consensus_queue.cc:1035] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:09.162132 18449 mvcc.cc:205] Tried to move back new op lower bound from 7334329012874022912 to 7334329012708884480. Current Snapshot: MvccSnapshot[applied={T|T < 7334329012874022912}]
I20260928 15:01:09.162503 18450 mvcc.cc:205] Tried to move back new op lower bound from 7334329012874022912 to 7334329012708884480. Current Snapshot: MvccSnapshot[applied={T|T < 7334329012874022912}]
I20260928 15:01:09.164796 17966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:09.164891 17966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:09.165701 17966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Sending DeleteTablet for 3 replicas of tablet 98c1d790e675420f85c95d6592151a9e
I20260928 15:01:09.166105 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 18006
I20260928 15:01:09.166092 18362 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c1d790e675420f85c95d6592151a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:09 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:34356
I20260928 15:01:09.166086 18229 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c1d790e675420f85c95d6592151a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:09 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:43730
I20260928 15:01:09.167050 18460 tablet_replica.cc:330] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: stopping tablet replica
I20260928 15:01:09.167152 18460 raft_consensus.cc:2229] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:09.167253 18460 raft_consensus.cc:2258] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:09.167758 18460 ts_tablet_manager.cc:1905] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:09.167953 18459 tablet_replica.cc:330] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: stopping tablet replica
I20260928 15:01:09.168045 18459 raft_consensus.cc:2229] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:09.168154 18459 raft_consensus.cc:2258] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:09.168591 18459 ts_tablet_manager.cc:1905] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:09.169904 18460 ts_tablet_manager.cc:1918] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:09.170001 18460 log.cc:1196] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/98c1d790e675420f85c95d6592151a9e
I20260928 15:01:09.170300 18460 ts_tablet_manager.cc:1939] T 98c1d790e675420f85c95d6592151a9e P df4a4d7b39e44ea2bf509e56976b68c9: Deleting consensus metadata
I20260928 15:01:09.170430 18459 ts_tablet_manager.cc:1918] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:09.170502 18459 log.cc:1196] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/98c1d790e675420f85c95d6592151a9e
I20260928 15:01:09.170765 17951 catalog_manager.cc:4841] TS df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): tablet 98c1d790e675420f85c95d6592151a9e (table test-table [id=f0435bf43e1449158a44783a20e3a7eb]) successfully deleted
I20260928 15:01:09.170801 18459 ts_tablet_manager.cc:1939] T 98c1d790e675420f85c95d6592151a9e P 158ac11269ca4836b585074888c55e9e: Deleting consensus metadata
I20260928 15:01:09.171171 17953 catalog_manager.cc:4841] TS 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): tablet 98c1d790e675420f85c95d6592151a9e (table test-table [id=f0435bf43e1449158a44783a20e3a7eb]) successfully deleted
W20260928 15:01:09.172739 17954 connection.cc:535] client connection to 127.1.252.193:36393 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 15:01:09.172833 17954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:01:09.172999 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 18161
I20260928 15:01:09.175850 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:09.175704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36393 (local address 127.0.0.1:48184)
0928 15:01:09.175776 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:09.175820 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:36393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:01:09.176024 17954 catalog_manager.cc:4584] TS 82d4474c73c14cae86ef50fb12686a58 (127.1.252.193:36393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98c1d790e675420f85c95d6592151a9e: Network error: Client connection negotiation failed: client connection to 127.1.252.193:36393: connect: Connection refused (error 111)
I20260928 15:01:09.180078 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 18299
I20260928 15:01:09.185364 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 17934
I20260928 15:01:29.191396 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:36033
--webserver_interface=127.1.252.254
--webserver_port=37727
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:29.272303 18462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:29.272526 18462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:29.272548 18462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:29.273768 18462 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:01:29.273813 18462 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:01:29.273828 18462 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:01:29.273839 18462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:29.273854 18462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:29.273866 18462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:29.273921 18462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:29.273938 18462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:29.275437 18462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:29.275715 18462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:29.276002 18462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:29.281239 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.281319 18462 server_base.cc:1034] running on GCE node
W20260928 15:01:29.281353 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.281481 18470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.281761 18462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:29.282065 18462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:29.283219 18462 hybrid_clock.cc:648] HybridClock initialized: now 1790607689283157 us; error 76 us; skew 500 ppm
Sep 28 15:01:29 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:29.285945 18462 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:01:29.286415 18462 webserver.cc:460] Webserver started at http://127.1.252.254:37727/ using document root <none> and password file <none>
I20260928 15:01:29.286652 18462 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:29.286726 18462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:29.288185 18462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:01:29.288888 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.289089 18462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:29.289196 18462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "7c41c0c8f9604f75862b9a0ed05017d8"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "41e35cb3f26e30be94235990c77c8e3d"
server_key_iv: "efbf196ff05acf5a234c9f0df313522c"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.289556 18462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:29.313860 18462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:29.316241 18462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:29.316416 18462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:29.320482 18462 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:36033
I20260928 15:01:29.320516 18529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:36033 every 8 connection(s)
I20260928 15:01:29.320891 18462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:29.322309 18530 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:01:29.322889 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap starting.
I20260928 15:01:29.326773 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18462
I20260928 15:01:29.327108 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:36393
--local_ip_for_outbound_sockets=127.1.252.193
--tserver_master_addrs=127.1.252.254:36033
--webserver_port=35281
--webserver_interface=127.1.252.193
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:01:29.331804 18530 log.cc:824] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:29.334038 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:01:29.334354 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Bootstrap complete.
I20260928 15:01:29.336637 18530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:29.336853 18530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c41c0c8f9604f75862b9a0ed05017d8, State: Initialized, Role: FOLLOWER
I20260928 15:01:29.337023 18530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:29.337090 18530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:29.337158 18530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:29.337201 18530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:01:29.338176 18530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:29.338259 18530 leader_election.cc:302] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c41c0c8f9604f75862b9a0ed05017d8; no voters:
I20260928 15:01:29.338441 18530 leader_election.cc:288] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:01:29.338496 18534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:01:29.338644 18534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [term 3 LEADER]: Becoming Leader. State: Replica: 7c41c0c8f9604f75862b9a0ed05017d8, State: Running, Role: LEADER
I20260928 15:01:29.338699 18530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:29.338742 18534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } }
I20260928 15:01:29.339012 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c41c0c8f9604f75862b9a0ed05017d8. Latest consensus state: current_term: 3 leader_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } } }
I20260928 15:01:29.339208 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:29.339015 18535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c41c0c8f9604f75862b9a0ed05017d8" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 36033 } } }
I20260928 15:01:29.339471 18545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:29.339505 18535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:29.340243 18545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:29.340341 18545 catalog_manager.cc:1259] Loaded cluster ID: ff7dff5a5fd04852be0a91d97695c07d
I20260928 15:01:29.340370 18545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:29.341570 18545 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:29.341745 18545 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Loaded TSK: 0
I20260928 15:01:29.341790 18545 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c41c0c8f9604f75862b9a0ed05017d8: Loaded TSK: 1
F20260928 15:01:29.342015 18545 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb931e09ad google::LogMessage::Fail() at ??:0
@ 0x7fdb931e388a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb931e04de google::LogMessage::Flush() at ??:0
@ 0x7fdb931e1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb9594d104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdb9594ad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdb9b13a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdb9b13a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdb9b13a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fdb9b100c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdb9b10aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb950ae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb950a3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb95f076db start_thread at ??:0
@ 0x7fdb91cde61f clone at ??:0
W20260928 15:01:29.427373 18532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:29.427615 18532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:29.427654 18532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:29.428859 18532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:29.428925 18532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:29.428967 18532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:29.430403 18532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:36393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=35281
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:29.430668 18532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:29.430895 18532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:29.435950 18573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.436065 18532 server_base.cc:1034] running on GCE node
W20260928 15:01:29.435982 18572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.436105 18575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.436447 18532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:29.436746 18532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:29.437933 18532 hybrid_clock.cc:648] HybridClock initialized: now 1790607689437904 us; error 42 us; skew 500 ppm
Sep 28 15:01:29 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:29.441120 18532 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:01:29.441733 18532 webserver.cc:460] Webserver started at http://127.1.252.193:35281/ using document root <none> and password file <none>
I20260928 15:01:29.441996 18532 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:29.442063 18532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:29.443925 18532 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:01:29.446090 18582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.446290 18532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:01:29.446401 18532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "82d4474c73c14cae86ef50fb12686a58"
format_stamp: "Formatted at 2026-09-28 15:00:47 on dist-test-slave-sf70"
server_key: "a3a03ffc6c4630b07c09ef561e606ee4"
server_key_iv: "00496eb8f294be6aab80f15049382acb"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.446799 18532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:29.458221 18532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:29.460635 18532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:29.460788 18532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:29.461030 18532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:29.461496 18589 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:01:29.463471 18532 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:01:29.463542 18532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:01:29.463590 18532 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:01:29.464334 18532 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:01:29.464390 18532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:29.464480 18589 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Bootstrap starting.
I20260928 15:01:29.471915 18532 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:36393
I20260928 15:01:29.472393 18532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:29.475423 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18532
I20260928 15:01:29.475950 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:36659
--local_ip_for_outbound_sockets=127.1.252.194
--tserver_master_addrs=127.1.252.254:36033
--webserver_port=44763
--webserver_interface=127.1.252.194
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:01:29.476099 18589 log.cc:824] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:29.476887 18696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:36393 every 8 connection(s)
I20260928 15:01:29.481884 18589 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:01:29.482395 18589 tablet_bootstrap.cc:492] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Bootstrap complete.
I20260928 15:01:29.483691 18589 ts_tablet_manager.cc:1397] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.009s
I20260928 15:01:29.485224 18589 raft_consensus.cc:348] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:29.485556 18589 raft_consensus.cc:729] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d4474c73c14cae86ef50fb12686a58, State: Initialized, Role: FOLLOWER
I20260928 15:01:29.485728 18589 consensus_queue.cc:260] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:29.486022 18589 ts_tablet_manager.cc:1428] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
W20260928 15:01:29.550942 18700 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:01:29.477516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:36471)
0928 15:01:29.478215 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.478330 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:29.479427 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:29.550723 (+ 71296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":94,"threads_started":1}
W20260928 15:01:29.551216 18697 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:36033 (0 consecutive failures): Network error: Failed to ping master at 127.1.252.254:36033: Client connection negotiation failed: client connection to 127.1.252.254:36033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:01:29.551576 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.551442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:53151)
0928 15:01:29.551500 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.551547 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 15:01:29.551964 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.551845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:56231)
0928 15:01:29.551913 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.551942 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:01:29.552084 18697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:01:29.563175 18701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:29.563360 18701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:29.563383 18701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:29.564553 18701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:29.564591 18701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:29.564617 18701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:29.565979 18701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:36659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=44763
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:29.566160 18701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:29.566327 18701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:01:29.571357 18701 server_base.cc:1034] running on GCE node
W20260928 15:01:29.571316 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.571305 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.571305 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.571715 18701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:29.571970 18701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:29.573136 18701 hybrid_clock.cc:648] HybridClock initialized: now 1790607689573120 us; error 29 us; skew 500 ppm
Sep 28 15:01:29 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:29.575686 18701 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:01:29.576179 18701 webserver.cc:460] Webserver started at http://127.1.252.194:44763/ using document root <none> and password file <none>
I20260928 15:01:29.576422 18701 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:29.576467 18701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:29.577903 18701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:29.578567 18717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.578799 18701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:29.578913 18701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "df4a4d7b39e44ea2bf509e56976b68c9"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "a14ca150085d3289d5191d5159c31fd2"
server_key_iv: "6f004b8a3960758449cef21c95a112b2"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.579316 18701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:29.591392 18701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:29.593721 18701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:29.593889 18701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:29.594180 18701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:29.594491 18701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:29.594544 18701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.594578 18701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:29.594607 18701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.600695 18701 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:36659
I20260928 15:01:29.600750 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:36659 every 8 connection(s)
I20260928 15:01:29.601114 18701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:29.602327 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.601838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:43339)
0928 15:01:29.602013 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.602146 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
W20260928 15:01:29.602756 18832 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:36033 (0 consecutive failures): Network error: Failed to ping master at 127.1.252.254:36033: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
I20260928 15:01:29.603101 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.602963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:43789)
0928 15:01:29.603047 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.603079 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:01:29.603392 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.603336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:45685)
0928 15:01:29.603358 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.603377 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 15:01:29.603552 18832 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:01:29.603622 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18701
I20260928 15:01:29.603917 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:36555
--local_ip_for_outbound_sockets=127.1.252.195
--tserver_master_addrs=127.1.252.254:36033
--webserver_port=39663
--webserver_interface=127.1.252.195
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:29.686228 18835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:29.686475 18835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:29.686514 18835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:29.687836 18835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:29.687953 18835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:29.688000 18835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:29.689723 18835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:36555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=39663
--tserver_master_addrs=127.1.252.254:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:29.690029 18835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:29.690282 18835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:29.695529 18841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.695528 18843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:29.695729 18840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:29.695744 18835 server_base.cc:1034] running on GCE node
I20260928 15:01:29.696059 18835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:29.696370 18835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:29.697551 18835 hybrid_clock.cc:648] HybridClock initialized: now 1790607689697504 us; error 59 us; skew 500 ppm
Sep 28 15:01:29 dist-test-slave-sf70 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:29.700243 18835 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:01:29.700703 18835 webserver.cc:460] Webserver started at http://127.1.252.195:39663/ using document root <none> and password file <none>
I20260928 15:01:29.700930 18835 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:29.700980 18835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:29.702378 18835 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:29.703107 18850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.703317 18835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:29.703392 18835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "158ac11269ca4836b585074888c55e9e"
format_stamp: "Formatted at 2026-09-28 15:00:48 on dist-test-slave-sf70"
server_key: "7e0c56f1e03513ad9ef94bcfa33c6afc"
server_key_iv: "862897f6121e1efa86ffae727dca2172"
server_key_version: "encryptionkey@0"
I20260928 15:01:29.703806 18835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:29.728714 18835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:29.730923 18835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:29.731106 18835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:29.731364 18835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:29.731703 18835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:29.731760 18835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.731797 18835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:29.731827 18835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:29.737895 18835 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:36555
I20260928 15:01:29.737937 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:36555 every 8 connection(s)
I20260928 15:01:29.738315 18835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:29.738905 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 18835
I20260928 15:01:29.739666 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.739187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:46687)
0928 15:01:29.739403 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.739495 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260928 15:01:29.740125 18965 heartbeater.cc:644] Failed to heartbeat to 127.1.252.254:36033 (0 consecutive failures): Network error: Failed to ping master at 127.1.252.254:36033: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
I20260928 15:01:29.740410 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.740316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:32775)
0928 15:01:29.740367 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.740392 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 15:01:29.740686 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:29.740629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:37087)
0928 15:01:29.740653 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:29.740673 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 15:01:29.740800 18965 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:01:30.552634 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.552407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:46561)
0928 15:01:30.552535 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.552570 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260928 15:01:30.604096 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.603847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:38087)
0928 15:01:30.604017 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.604048 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260928 15:01:30.741312 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.741103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:49995)
0928 15:01:30.741229 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.741261 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260928 15:01:30.767491 18972 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:30.767617 18972 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:30.768002 18972 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
Sep 28 15:01:30 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.195@KRBTEST.COM
Sep 28 15:01:30 dist-test-slave-sf70 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.194@KRBTEST.COM
I20260928 15:01:30.787217 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.768265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:32955 (local address 127.1.252.195:36555)
0928 15:01:30.768386 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:30.768400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:30.768435 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:01:30.769035 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:30.769037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:30.769093 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:30.769247 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:30.770332 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.771753 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.773384 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.774017 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.774135 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.783769 (+ 9634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:30.783773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:30.783773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:30.783785 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:30.785519 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:30.785974 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:30.785979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:30.785979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:30.786071 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:30.786337 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:30.786338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:30.786338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:30.786584 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:30.786670 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:30.786971 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:01:30.787101 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
I20260928 15:01:30.787206 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.768141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:36555 (local address 127.1.252.193:32955)
0928 15:01:30.768197 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.768209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:30.768266 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:30.769336 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:30.769339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:30.769873 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:30.770188 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.770195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.771902 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.771903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.773280 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.773282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.774126 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.774127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.774276 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.781635 (+ 7359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:30.781639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:30.783616 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:30.785624 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:30.785631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:30.785633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:30.785861 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:30.786152 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:30.786153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:30.786153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:30.786262 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:30.786663 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:30.786670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:30.786803 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 15:01:30.786910 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:01:30.787099 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":108}
I20260928 15:01:30.787331 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.768192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:36659 (local address 127.1.252.193:35199)
0928 15:01:30.768348 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:30.768359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:30.768412 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:30.769336 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:30.769339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:30.769873 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:30.770188 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.770195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.771927 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.771927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.773280 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:30.773282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.774126 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.774127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:30.774276 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.781534 (+ 7258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:30.781552 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:30.783958 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:30.785955 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:30.785956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:30.785956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:30.786137 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:30.786405 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:30.786406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:30.786406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:30.786482 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:30.786860 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:30.786861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:30.786909 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:01:30.786995 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:01:30.787188 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":150,"thread_start_us":73,"threads_started":1}
I20260928 15:01:30.788024 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:30.768307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:35199 (local address 127.1.252.194:36659)
0928 15:01:30.768459 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:30.768467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:30.768494 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:01:30.769029 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:30.769033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:30.769100 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:30.769247 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:30.770332 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.771753 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.773384 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:30.774017 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:30.774137 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:30.784083 (+ 9946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:30.784087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:30.784087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:30.784097 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:30.785840 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:30.786230 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:30.786233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:30.786233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:30.786326 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:30.786546 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:30.786547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:30.786547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:30.786780 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:30.786864 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:30.787055 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 15:01:30.787210 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
W20260928 15:01:30.788239 18585 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
W20260928 15:01:30.788681 18583 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
I20260928 15:01:30.788750 18583 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58; no voters: 158ac11269ca4836b585074888c55e9e, df4a4d7b39e44ea2bf509e56976b68c9
I20260928 15:01:30.788880 18972 raft_consensus.cc:2726] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:01:31.553359 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.553137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:42659)
0928 15:01:31.553270 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.553310 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 15:01:31.604776 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.604575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:53157)
0928 15:01:31.604698 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.604736 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260928 15:01:31.741937 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:31.741746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:49075)
0928 15:01:31.741866 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:31.741900 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260928 15:01:32.330137 18979 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:32.330231 18979 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:32.330433 18979 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
W20260928 15:01:32.330962 18583 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
W20260928 15:01:32.331028 18585 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
I20260928 15:01:32.331055 18583 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58; no voters: 158ac11269ca4836b585074888c55e9e, df4a4d7b39e44ea2bf509e56976b68c9
I20260928 15:01:32.331144 18979 raft_consensus.cc:2726] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:01:32.554061 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:32.553838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:46307)
0928 15:01:32.553977 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:32.554013 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 15:01:32.605646 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:32.605425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:58271)
0928 15:01:32.605560 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:32.605604 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260928 15:01:32.742703 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:32.742436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:41969)
0928 15:01:32.742598 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:32.742658 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":61,"threads_started":1}
I20260928 15:01:33.554811 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:33.554560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:33737)
0928 15:01:33.554719 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:33.554762 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260928 15:01:33.606345 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:33.606129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:46797)
0928 15:01:33.606263 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:33.606303 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 15:01:33.743379 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:33.743176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:45951)
0928 15:01:33.743306 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:33.743341 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260928 15:01:34.314889 18986 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:34.314990 18986 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:34.315188 18986 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
W20260928 15:01:34.315512 18585 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
W20260928 15:01:34.315666 18583 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
I20260928 15:01:34.315701 18583 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58; no voters: 158ac11269ca4836b585074888c55e9e, df4a4d7b39e44ea2bf509e56976b68c9
I20260928 15:01:34.315766 18986 raft_consensus.cc:2726] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:01:34.555703 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:34.555376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:48871)
0928 15:01:34.555555 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:34.555642 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
I20260928 15:01:34.607163 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:34.606956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:47441)
0928 15:01:34.607079 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:34.607122 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260928 15:01:34.744068 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:34.743788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:43053)
0928 15:01:34.743988 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:34.744026 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":119,"threads_started":1}
I20260928 15:01:35.556316 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:35.556107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:51223)
0928 15:01:35.556234 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:35.556268 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 15:01:35.607836 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:35.607648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:59683)
0928 15:01:35.607763 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:35.607797 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260928 15:01:35.744762 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:35.744507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:35799)
0928 15:01:35.744663 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:35.744707 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 15:01:36.557010 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:36.556792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:37703)
0928 15:01:36.556928 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:36.556964 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260928 15:01:36.608657 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:36.608403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:45345)
0928 15:01:36.608549 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:36.608613 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260928 15:01:36.745447 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:36.745236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:39207)
0928 15:01:36.745363 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:36.745402 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260928 15:01:37.530814 18996 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:37.530975 18996 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:37.531198 18996 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
W20260928 15:01:37.531630 18583 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
W20260928 15:01:37.531716 18585 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
I20260928 15:01:37.531756 18585 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58; no voters: 158ac11269ca4836b585074888c55e9e, df4a4d7b39e44ea2bf509e56976b68c9
I20260928 15:01:37.531826 18996 raft_consensus.cc:2726] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:01:37.557781 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:37.557546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:55603)
0928 15:01:37.557696 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:37.557733 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260928 15:01:37.609372 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:37.609152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:46389)
0928 15:01:37.609285 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:37.609316 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260928 15:01:37.746196 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:37.745958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:57747)
0928 15:01:37.746099 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:37.746135 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 15:01:38.558477 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:38.558240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:54253)
0928 15:01:38.558391 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:38.558428 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260928 15:01:38.610087 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:38.609862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:43161)
0928 15:01:38.609995 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:38.610039 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260928 15:01:38.746904 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:38.746695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:50501)
0928 15:01:38.746824 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:38.746859 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260928 15:01:39.559269 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:39.559018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:37311)
0928 15:01:39.559175 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:39.559219 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 15:01:39.611002 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:39.610773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:58881)
0928 15:01:39.610913 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:39.610954 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260928 15:01:39.747586 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:39.747346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:39033)
0928 15:01:39.747462 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:39.747499 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260928 15:01:40.560026 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:40.559719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:60075)
0928 15:01:40.559871 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:40.559959 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260928 15:01:40.611696 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:40.611485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:60637)
0928 15:01:40.611612 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:40.611649 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260928 15:01:40.748263 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:40.748024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:37059)
0928 15:01:40.748178 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:40.748216 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 15:01:41.560763 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:41.560528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:45537)
0928 15:01:41.560679 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:41.560715 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260928 15:01:41.612385 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:41.612143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:34033)
0928 15:01:41.612297 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:41.612335 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260928 15:01:41.748970 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:41.748735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:60375)
0928 15:01:41.748883 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:41.748921 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260928 15:01:41.955195 19012 raft_consensus.cc:482] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:41.955338 19012 raft_consensus.cc:504] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158ac11269ca4836b585074888c55e9e" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 36555 } } peers { permanent_uuid: "df4a4d7b39e44ea2bf509e56976b68c9" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 36659 } } peers { permanent_uuid: "82d4474c73c14cae86ef50fb12686a58" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36393 } }
I20260928 15:01:41.955569 19012 leader_election.cc:288] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555), df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659)
W20260928 15:01:41.956043 18585 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 158ac11269ca4836b585074888c55e9e (127.1.252.195:36555): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
W20260928 15:01:41.956104 18583 leader_election.cc:341] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df4a4d7b39e44ea2bf509e56976b68c9 (127.1.252.194:36659): Not found: Tablet not found: 98c1d790e675420f85c95d6592151a9e
I20260928 15:01:41.956137 18583 leader_election.cc:302] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d4474c73c14cae86ef50fb12686a58; no voters: 158ac11269ca4836b585074888c55e9e, df4a4d7b39e44ea2bf509e56976b68c9
I20260928 15:01:41.956274 19012 raft_consensus.cc:2726] T 98c1d790e675420f85c95d6592151a9e P 82d4474c73c14cae86ef50fb12686a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:01:42.561560 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:42.561283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:46199)
0928 15:01:42.561426 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:42.561511 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260928 15:01:42.613191 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:42.612939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:34213)
0928 15:01:42.613097 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:42.613144 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260928 15:01:42.749732 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:42.749504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:34243)
0928 15:01:42.749640 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:42.749684 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 15:01:43.562373 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:43.562138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:50053)
0928 15:01:43.562287 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:43.562326 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260928 15:01:43.614001 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:43.613749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:52999)
0928 15:01:43.613904 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:43.613953 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20260928 15:01:43.750797 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:43.750552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.195:34349)
0928 15:01:43.750689 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:43.750735 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260928 15:01:44.563238 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:44.562916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.193:47971)
0928 15:01:44.563122 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:44.563178 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260928 15:01:44.614863 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:44.614627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:36033 (local address 127.1.252.194:53719)
0928 15:01:44.614775 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:44.614810 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.254:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260928 15:01:44.741303 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 18532
I20260928 15:01:44.746153 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 18701
I20260928 15:01:44.751490 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 18835
2026-09-28T15:01:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58312 ms)
[----------] 30 tests from SecurityITest (230527 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:44 dist-test-slave-sf70 krb5kdc[19024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:44 dist-test-slave-sf70 krb5kdc[19024](info): set up 2 sockets
Sep 28 15:01:44 dist-test-slave-sf70 krb5kdc[19024](info): commencing operation
krb5kdc: starting...
W20260928 15:01:45.845938 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:45 dist-test-slave-sf70 krb5kdc[19024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:01:45.966943 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37497
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:37497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:46.047170 19037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.047415 19037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.047451 19037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.048600 19037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:46.048674 19037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.048697 19037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.048718 19037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:46.048746 19037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:46.050230 19037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:37497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:37497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:46.050506 19037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.050729 19037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:46.055964 19046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.055995 19044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.056037 19043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.056154 19037 server_base.cc:1034] running on GCE node
I20260928 15:01:46.056412 19037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.056739 19037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.057915 19037 hybrid_clock.cc:648] HybridClock initialized: now 1790607706057886 us; error 44 us; skew 500 ppm
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:46.060598 19037 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:01:46.061055 19037 webserver.cc:460] Webserver started at http://127.1.252.254:43081/ using document root <none> and password file <none>
I20260928 15:01:46.061305 19037 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.061378 19037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.061518 19037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.062602 19037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "c8739d6942f44b67963f38c0b338857d"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "7faffc33951d7eb088a75eb4208b040e"
server_key_iv: "4b1cee5c1b630c98e3be76b9dc239d58"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.062970 19037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "c8739d6942f44b67963f38c0b338857d"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "7faffc33951d7eb088a75eb4208b040e"
server_key_iv: "4b1cee5c1b630c98e3be76b9dc239d58"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.064395 19037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:46.065163 19053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.065338 19037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:46.065443 19037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "c8739d6942f44b67963f38c0b338857d"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "7faffc33951d7eb088a75eb4208b040e"
server_key_iv: "4b1cee5c1b630c98e3be76b9dc239d58"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.065521 19037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:46.080749 19037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:46.083134 19037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:46.083299 19037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:46.087113 19037 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:37497
I20260928 15:01:46.087168 19105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:37497 every 8 connection(s)
I20260928 15:01:46.087502 19037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:46.088465 19106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:46.090878 19106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d: Bootstrap starting.
I20260928 15:01:46.091478 19106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:46.091780 19106 log.cc:824] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:46.091853 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19037
I20260928 15:01:46.092017 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:46.092275 2035 external_mini_cluster.cc:1428] Setting key 5585d619bf37549aa28d749e0aa12e24
I20260928 15:01:46.092801 19106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d: No bootstrap required, opened a new log
I20260928 15:01:46.094969 19106 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8739d6942f44b67963f38c0b338857d" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37497 } }
I20260928 15:01:46.095153 19106 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:46.095187 19106 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8739d6942f44b67963f38c0b338857d, State: Initialized, Role: FOLLOWER
I20260928 15:01:46.095350 19106 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8739d6942f44b67963f38c0b338857d" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37497 } }
I20260928 15:01:46.095441 19106 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:46.095471 19106 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:46.095508 19106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:46.096247 19106 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8739d6942f44b67963f38c0b338857d" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37497 } }
I20260928 15:01:46.096351 19106 leader_election.cc:302] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8739d6942f44b67963f38c0b338857d; no voters:
I20260928 15:01:46.096505 19106 leader_election.cc:288] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:46.096580 19111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:46.096743 19106 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:46.096784 19111 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [term 1 LEADER]: Becoming Leader. State: Replica: c8739d6942f44b67963f38c0b338857d, State: Running, Role: LEADER
I20260928 15:01:46.096966 19111 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8739d6942f44b67963f38c0b338857d" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37497 } }
I20260928 15:01:46.097366 19112 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8739d6942f44b67963f38c0b338857d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8739d6942f44b67963f38c0b338857d" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37497 } } }
I20260928 15:01:46.097447 19112 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:46.097678 19113 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8739d6942f44b67963f38c0b338857d. Latest consensus state: current_term: 1 leader_uuid: "c8739d6942f44b67963f38c0b338857d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8739d6942f44b67963f38c0b338857d" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 37497 } } }
I20260928 15:01:46.098059 19113 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:46.097992 19120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:46.098827 19120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:46.100574 19120 catalog_manager.cc:1347] Generated new cluster ID: 85d708c3ca624633a919cab28fecb0ad
I20260928 15:01:46.100625 19120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:46.105336 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.092854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.1.252.254:37497)
0928 15:01:46.093068 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.093077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.093112 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:01:46.094159 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.094169 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.094229 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.094377 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.095000 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.096779 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.097973 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.098617 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.098742 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.101549 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.101553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.101553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.101565 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.103541 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.103942 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.103946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.103946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.104053 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.104257 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.104258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.104259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.104630 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.104794 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.104872 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.105116 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260928 15:01:46.118623 19120 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:46.119318 19120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:46.129902 19120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d: Generated new TSK 0
I20260928 15:01:46.130120 19120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 15:01:46.167129 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37497
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:46.271212 19132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.271407 19132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.271428 19132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.272853 19132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.272904 19132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.272931 19132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:46.274360 19132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:46.274607 19132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.274830 19132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:46.280267 19138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.280297 19140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.280526 19137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.280758 19132 server_base.cc:1034] running on GCE node
I20260928 15:01:46.280999 19132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.281298 19132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.282449 19132 hybrid_clock.cc:648] HybridClock initialized: now 1790607706282432 us; error 25 us; skew 500 ppm
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:46.285826 19132 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:01:46.286388 19132 webserver.cc:460] Webserver started at http://127.1.252.193:33635/ using document root <none> and password file <none>
I20260928 15:01:46.286664 19132 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.286731 19132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.286865 19132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.287988 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "aba40cbbf8276a4cb6ee7ec63fb37c9e"
server_key_iv: "3c06e87fed01d2c732c194e3451ca917"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.288321 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "aba40cbbf8276a4cb6ee7ec63fb37c9e"
server_key_iv: "3c06e87fed01d2c732c194e3451ca917"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.289614 19132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:01:46.290336 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.290716 19132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:01:46.290791 19132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "aba40cbbf8276a4cb6ee7ec63fb37c9e"
server_key_iv: "3c06e87fed01d2c732c194e3451ca917"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.290881 19132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:46.301642 19132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:46.303710 19132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:46.303857 19132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:46.304148 19132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:46.304489 19132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:46.304524 19132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.304566 19132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:46.304591 19132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.310405 19132 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:38507
I20260928 15:01:46.310482 19261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:38507 every 8 connection(s)
I20260928 15:01:46.310783 19132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:46.313040 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19132
I20260928 15:01:46.313131 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:46.313325 2035 external_mini_cluster.cc:1428] Setting key 818e2691d20d40669cc454ec159956b4
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:46.321223 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.311478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:44255 (local address 127.1.252.254:37497)
0928 15:01:46.311535 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.311537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.311810 (+ 273us) server_negotiation.cc:407] Connection header received
0928 15:01:46.312364 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.312366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.312376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.312424 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.313444 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.314218 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.315350 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.315618 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.315676 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.317967 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.317969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.317969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.317977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.319675 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.320135 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.320137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.320137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.320200 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.320488 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.320489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.320490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.320642 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.320719 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.321026 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.321118 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 15:01:46.321250 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.311529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37497 (local address 127.1.252.193:44255)
0928 15:01:46.311690 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.311726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.312276 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.312486 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.312489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.313067 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:46.313322 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.313327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.314343 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.314344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.315230 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.315233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.315722 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.315723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.315896 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.316439 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:46.316449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:46.317873 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:46.319761 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:46.319767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:46.319769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:46.320066 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:46.320299 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:46.320300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:46.320300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:46.320407 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:46.320739 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:46.320745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:46.320872 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.320962 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.321087 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260928 15:01:46.321832 19262 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37497
I20260928 15:01:46.321985 19262 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:46.322180 19262 heartbeater.cc:511] Master 127.1.252.254:37497 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 15:01:46.322688 19070 ts_manager.cc:194] Registered new tserver with Master: 9aaee8ee1a06479b831d6a0d0b04e7e0 (127.1.252.193:38507)
I20260928 15:01:46.323355 19070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:44255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 15:01:46.341219 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37497
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:46.424934 19267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.425176 19267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.425201 19267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.426470 19267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.426515 19267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.426549 19267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:46.427999 19267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:46.428246 19267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.428485 19267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:46.433712 19275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.433722 19273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.433892 19272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.434126 19267 server_base.cc:1034] running on GCE node
I20260928 15:01:46.434337 19267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.434592 19267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.435758 19267 hybrid_clock.cc:648] HybridClock initialized: now 1790607706435757 us; error 42 us; skew 500 ppm
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:46.438344 19267 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:01:46.438818 19267 webserver.cc:460] Webserver started at http://127.1.252.194:33209/ using document root <none> and password file <none>
I20260928 15:01:46.439067 19267 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.439136 19267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.439271 19267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.440390 19267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "8a3d117d0422451ca158369566740a3a"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "09cb84db983e9c18ba7b39371da5023d"
server_key_iv: "48ba4e3330f6ed470969eb242ff845dc"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.440752 19267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "8a3d117d0422451ca158369566740a3a"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "09cb84db983e9c18ba7b39371da5023d"
server_key_iv: "48ba4e3330f6ed470969eb242ff845dc"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.442243 19267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:01:46.442972 19282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.443179 19267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:46.443277 19267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "8a3d117d0422451ca158369566740a3a"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "09cb84db983e9c18ba7b39371da5023d"
server_key_iv: "48ba4e3330f6ed470969eb242ff845dc"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.443359 19267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:46.461943 19267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:46.464053 19267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:46.464217 19267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:46.464458 19267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:46.464774 19267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:46.464833 19267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.464875 19267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:46.464898 19267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.471483 19267 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:38083
I20260928 15:01:46.471570 19396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:38083 every 8 connection(s)
I20260928 15:01:46.471925 19267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:46.476562 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19267
I20260928 15:01:46.476686 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:46.476886 2035 external_mini_cluster.cc:1428] Setting key 23e1aef1b214b6329051131d378f2817
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:46.481635 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.472741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:60121 (local address 127.1.252.254:37497)
0928 15:01:46.472786 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.472788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.473002 (+ 214us) server_negotiation.cc:407] Connection header received
0928 15:01:46.473478 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.473479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.473487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.473562 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.474450 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.474952 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.475991 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.476194 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.476244 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.478539 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.478541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.478541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.478548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.480253 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.480663 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.480664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.480665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.480722 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.480984 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.480985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.480985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.481105 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.481182 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.481456 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.481545 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:01:46.481839 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.472706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37497 (local address 127.1.252.194:60121)
0928 15:01:46.472876 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.472918 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.473397 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.473620 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.473624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.474184 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:46.474376 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.474380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.475031 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.475032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.475872 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.475875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.476269 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.476271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.476408 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.476944 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:46.476954 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:46.478433 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:46.480333 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:46.480341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:46.480343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:46.480573 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:46.480807 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:46.480808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:46.480808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:46.480916 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:46.481174 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:46.481181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:46.481323 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.481433 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.481597 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 15:01:46.482337 19397 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37497
I20260928 15:01:46.482476 19397 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:46.482687 19397 heartbeater.cc:511] Master 127.1.252.254:37497 requested a full tablet report, sending...
I20260928 15:01:46.483080 19070 ts_manager.cc:194] Registered new tserver with Master: 8a3d117d0422451ca158369566740a3a (127.1.252.194:38083)
I20260928 15:01:46.483444 19070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:60121
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 15:01:46.503382 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37497
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:46.584452 19402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.584656 19402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.584678 19402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.585755 19402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.585788 19402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.585813 19402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:46.587141 19402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:46.587347 19402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.587548 19402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:46.592734 19410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.592739 19408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.592751 19407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.592873 19402 server_base.cc:1034] running on GCE node
I20260928 15:01:46.593202 19402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.593503 19402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.594686 19402 hybrid_clock.cc:648] HybridClock initialized: now 1790607706594669 us; error 28 us; skew 500 ppm
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:46.597191 19402 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:01:46.597801 19402 webserver.cc:460] Webserver started at http://127.1.252.195:39217/ using document root <none> and password file <none>
I20260928 15:01:46.598099 19402 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.598184 19402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.598413 19402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.599504 19402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "125a56f60a014f6c9b517f77c4f5de18"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "d25b7b9e8141a363b30895dc78c3db35"
server_key_iv: "0fccd6665caae22f2797868ced94ff5f"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.599924 19402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "125a56f60a014f6c9b517f77c4f5de18"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "d25b7b9e8141a363b30895dc78c3db35"
server_key_iv: "0fccd6665caae22f2797868ced94ff5f"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.601311 19402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:46.602123 19417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.602322 19402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:46.602425 19402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "125a56f60a014f6c9b517f77c4f5de18"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "d25b7b9e8141a363b30895dc78c3db35"
server_key_iv: "0fccd6665caae22f2797868ced94ff5f"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.602537 19402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:46.609335 19402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:46.611207 19402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:46.611384 19402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:46.611624 19402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:46.612007 19402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:46.612061 19402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.612103 19402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:46.612130 19402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.617933 19402 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:41295
I20260928 15:01:46.618006 19530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:41295 every 8 connection(s)
I20260928 15:01:46.618315 19402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:01:46 dist-test-slave-sf70 krb5kdc[19024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:46.627168 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.619059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:33187 (local address 127.1.252.254:37497)
0928 15:01:46.619116 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.619118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.619308 (+ 190us) server_negotiation.cc:407] Connection header received
0928 15:01:46.619788 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.619789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.619800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.619852 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.620804 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.621328 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.622325 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.622542 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.622615 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.624756 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.624757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.624758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.624764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.625939 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.626324 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.626325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.626325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.626378 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.626622 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.626623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.626623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.626736 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.626792 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.627024 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.627086 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:01:46.627282 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.619004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:37497 (local address 127.1.252.195:33187)
0928 15:01:46.619187 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.619221 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.619700 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.619962 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.619966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.620533 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:46.620725 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.620729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.621415 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.621415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.622226 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.622228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.622627 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.622628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.622725 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.623210 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:46.623218 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:46.624671 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:46.626010 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:46.626016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:46.626018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:46.626252 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:46.626442 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:46.626442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:46.626443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:46.626543 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:46.626798 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:46.626802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:46.626914 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.626994 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.627122 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260928 15:01:46.627988 19531 heartbeater.cc:347] Connected to a master server at 127.1.252.254:37497
I20260928 15:01:46.628032 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19402
I20260928 15:01:46.628100 19531 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:46.628144 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:46.628285 19531 heartbeater.cc:511] Master 127.1.252.254:37497 requested a full tablet report, sending...
I20260928 15:01:46.628314 2035 external_mini_cluster.cc:1428] Setting key f87151b4ab6b89499922bff652e9f11f
I20260928 15:01:46.628759 19070 ts_manager.cc:194] Registered new tserver with Master: 125a56f60a014f6c9b517f77c4f5de18 (127.1.252.195:41295)
I20260928 15:01:46.629251 19070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:33187
I20260928 15:01:46.629293 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:46.636884 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.630728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.1.252.254:37497)
0928 15:01:46.630784 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.630785 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.630925 (+ 140us) server_negotiation.cc:407] Connection header received
0928 15:01:46.630973 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.630974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.630983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.631027 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.631533 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.632067 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.632879 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.633070 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.633125 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.634360 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.634362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.634362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:46.634368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.635534 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.635837 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.635837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.635838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.636035 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:46.636227 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:46.636227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:46.636228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:46.636365 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.636466 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.636664 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.636735 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 15:01:46.638509 19070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:46.640260 19070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:46.646961 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.642777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:38507 (local address 127.0.0.1:58852)
0928 15:01:46.642966 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.642973 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.643036 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.643393 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.643394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.643396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:46.643436 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.643437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.644833 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.644834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.645752 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.645754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.646688 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.646689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.646754 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.646773 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.646810 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.646873 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":6,"thread_start_us":63,"threads_started":1}
I20260928 15:01:46.647027 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.642696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.1.252.195:41295)
0928 15:01:46.643196 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.643202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.643213 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:01:46.643254 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.643256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.643273 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.643346 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:46.643600 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.644371 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.645862 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.646532 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.646627 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.646688 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.646879 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.646933 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":95,"threads_started":1}
I20260928 15:01:46.647311 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.642529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:38083 (local address 127.0.0.1:34030)
0928 15:01:46.642665 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.642708 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.642769 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.643291 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.643295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.643297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:46.643366 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.643368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.644809 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.644810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.645867 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.645869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.646761 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.646762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.646868 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.646872 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.646922 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.646961 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":11,"thread_start_us":45,"threads_started":1}
I20260928 15:01:46.647333 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.642902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.1.252.194:38083)
0928 15:01:46.643102 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.643107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.643118 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:01:46.643161 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.643163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.643179 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.643443 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:46.643521 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.644706 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.645947 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.646648 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.646740 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.646798 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.647220 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.647275 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260928 15:01:46.647495 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.643018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.1.252.193:38507)
0928 15:01:46.643173 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.643176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.643185 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:01:46.643221 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.643223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.643238 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.643509 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:46.643595 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.644705 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.645840 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.646582 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.647166 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.647230 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.647255 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.647298 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260928 15:01:46.646966 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.642545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:41295 (local address 127.0.0.1:59518)
0928 15:01:46.642739 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.642751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.642807 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.643425 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.643426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.643441 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:46.643509 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.643510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.644480 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.644481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.645762 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.645764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.646640 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.646641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.646747 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.646763 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.646833 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.646905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":26,"thread_start_us":96,"threads_started":1}
I20260928 15:01:46.648135 19196 tablet_service.cc:1467] Processing CreateTablet for tablet 55d9c1fa17c74ad7b2d2ed370e415299 (DEFAULT_TABLE table=test-table [id=76578218a83447e6b444e59955d27bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:46.648114 19331 tablet_service.cc:1467] Processing CreateTablet for tablet 55d9c1fa17c74ad7b2d2ed370e415299 (DEFAULT_TABLE table=test-table [id=76578218a83447e6b444e59955d27bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:46.648284 19465 tablet_service.cc:1467] Processing CreateTablet for tablet 55d9c1fa17c74ad7b2d2ed370e415299 (DEFAULT_TABLE table=test-table [id=76578218a83447e6b444e59955d27bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:46.648412 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d9c1fa17c74ad7b2d2ed370e415299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:46.648448 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d9c1fa17c74ad7b2d2ed370e415299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:46.648543 19465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d9c1fa17c74ad7b2d2ed370e415299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:46.651006 19548 tablet_bootstrap.cc:492] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: Bootstrap starting.
I20260928 15:01:46.651664 19548 tablet_bootstrap.cc:654] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:46.651751 19549 tablet_bootstrap.cc:492] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Bootstrap starting.
I20260928 15:01:46.651937 19550 tablet_bootstrap.cc:492] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Bootstrap starting.
I20260928 15:01:46.652289 19548 log.cc:824] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:46.652456 19549 tablet_bootstrap.cc:654] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:46.652556 19550 tablet_bootstrap.cc:654] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:46.652895 19549 log.cc:824] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:46.653103 19550 log.cc:824] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:46.653621 19548 tablet_bootstrap.cc:492] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: No bootstrap required, opened a new log
I20260928 15:01:46.653646 19549 tablet_bootstrap.cc:492] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: No bootstrap required, opened a new log
I20260928 15:01:46.653692 19548 ts_tablet_manager.cc:1397] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:01:46.653712 19549 ts_tablet_manager.cc:1397] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:46.653913 19550 tablet_bootstrap.cc:492] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: No bootstrap required, opened a new log
I20260928 15:01:46.653966 19550 ts_tablet_manager.cc:1397] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:46.655073 19549 raft_consensus.cc:348] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.655205 19549 raft_consensus.cc:374] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:46.655228 19549 raft_consensus.cc:729] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125a56f60a014f6c9b517f77c4f5de18, State: Initialized, Role: FOLLOWER
I20260928 15:01:46.655349 19549 consensus_queue.cc:260] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.655639 19549 ts_tablet_manager.cc:1428] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:46.655655 19531 heartbeater.cc:503] Master 127.1.252.254:37497 was elected leader, sending a full tablet report...
I20260928 15:01:46.655737 19548 raft_consensus.cc:348] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.655920 19548 raft_consensus.cc:374] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:46.655965 19548 raft_consensus.cc:729] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aaee8ee1a06479b831d6a0d0b04e7e0, State: Initialized, Role: FOLLOWER
I20260928 15:01:46.655929 19550 raft_consensus.cc:348] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.656107 19550 raft_consensus.cc:374] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:46.656090 19548 consensus_queue.cc:260] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.656152 19550 raft_consensus.cc:729] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3d117d0422451ca158369566740a3a, State: Initialized, Role: FOLLOWER
I20260928 15:01:46.656276 19550 consensus_queue.cc:260] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.656340 19548 ts_tablet_manager.cc:1428] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:01:46.656443 19262 heartbeater.cc:503] Master 127.1.252.254:37497 was elected leader, sending a full tablet report...
I20260928 15:01:46.656527 19550 ts_tablet_manager.cc:1428] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 15:01:46.656571 19397 heartbeater.cc:503] Master 127.1.252.254:37497 was elected leader, sending a full tablet report...
I20260928 15:01:46.664649 19556 raft_consensus.cc:482] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:46.664778 19556 raft_consensus.cc:504] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.665088 19556 leader_election.cc:288] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125a56f60a014f6c9b517f77c4f5de18 (127.1.252.195:41295), 9aaee8ee1a06479b831d6a0d0b04e7e0 (127.1.252.193:38507)
I20260928 15:01:46.669047 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.665578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:43575 (local address 127.1.252.193:38507)
0928 15:01:46.665621 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.665623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.665745 (+ 122us) server_negotiation.cc:407] Connection header received
0928 15:01:46.665788 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.665790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.665801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.665848 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:46.666075 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.666744 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.668068 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.668702 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.668762 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.668770 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.668942 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.668979 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:01:46.669050 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.665397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:44863 (local address 127.1.252.195:41295)
0928 15:01:46.665510 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.665512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.665522 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:01:46.665611 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.665612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.665622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.665673 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:46.665846 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.666637 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.668053 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.668680 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.668760 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.668777 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.668942 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.668995 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:01:46.669064 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.665339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:41295 (local address 127.1.252.194:44863)
0928 15:01:46.665423 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.665436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.665528 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.665677 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.665678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.665680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:46.665755 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.665757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.666724 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.666725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.667944 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.667947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.668774 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.668775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.668862 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.668865 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.668922 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.668976 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":12}
I20260928 15:01:46.669072 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.665469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:38507 (local address 127.1.252.194:43575)
0928 15:01:46.665626 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.665636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:46.665739 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:46.665920 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:46.665921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:46.665922 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:46.666010 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.666012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.666819 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.666820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.667996 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:46.667998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.668774 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.668775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:46.668855 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.668860 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:01:46.668938 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:01:46.668975 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260928 15:01:46.669533 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9c1fa17c74ad7b2d2ed370e415299" candidate_uuid: "8a3d117d0422451ca158369566740a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125a56f60a014f6c9b517f77c4f5de18" is_pre_election: true
I20260928 15:01:46.669611 19216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9c1fa17c74ad7b2d2ed370e415299" candidate_uuid: "8a3d117d0422451ca158369566740a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" is_pre_election: true
I20260928 15:01:46.669694 19485 raft_consensus.cc:2454] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3d117d0422451ca158369566740a3a in term 0.
I20260928 15:01:46.669723 19216 raft_consensus.cc:2454] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3d117d0422451ca158369566740a3a in term 0.
I20260928 15:01:46.669929 19286 leader_election.cc:302] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a3d117d0422451ca158369566740a3a, 9aaee8ee1a06479b831d6a0d0b04e7e0; no voters:
I20260928 15:01:46.670044 19556 raft_consensus.cc:2781] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:46.670114 19556 raft_consensus.cc:482] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:46.670138 19556 raft_consensus.cc:3037] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:46.670819 19556 raft_consensus.cc:504] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.670955 19556 leader_election.cc:288] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [CANDIDATE]: Term 1 election: Requested vote from peers 125a56f60a014f6c9b517f77c4f5de18 (127.1.252.195:41295), 9aaee8ee1a06479b831d6a0d0b04e7e0 (127.1.252.193:38507)
I20260928 15:01:46.671166 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9c1fa17c74ad7b2d2ed370e415299" candidate_uuid: "8a3d117d0422451ca158369566740a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125a56f60a014f6c9b517f77c4f5de18"
I20260928 15:01:46.671183 19216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9c1fa17c74ad7b2d2ed370e415299" candidate_uuid: "8a3d117d0422451ca158369566740a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0"
I20260928 15:01:46.671262 19485 raft_consensus.cc:3037] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:46.671262 19216 raft_consensus.cc:3037] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:46.672204 19485 raft_consensus.cc:2454] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3d117d0422451ca158369566740a3a in term 1.
I20260928 15:01:46.672204 19216 raft_consensus.cc:2454] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3d117d0422451ca158369566740a3a in term 1.
I20260928 15:01:46.672463 19286 leader_election.cc:302] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a3d117d0422451ca158369566740a3a, 9aaee8ee1a06479b831d6a0d0b04e7e0; no voters:
I20260928 15:01:46.672581 19556 raft_consensus.cc:2781] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:46.672734 19556 raft_consensus.cc:686] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 1 LEADER]: Becoming Leader. State: Replica: 8a3d117d0422451ca158369566740a3a, State: Running, Role: LEADER
I20260928 15:01:46.672843 19556 consensus_queue.cc:237] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } }
I20260928 15:01:46.673727 19070 catalog_manager.cc:5495] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a3d117d0422451ca158369566740a3a (127.1.252.194). New cstate: current_term: 1 leader_uuid: "8a3d117d0422451ca158369566740a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d117d0422451ca158369566740a3a" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 38083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260928 15:01:46.681706 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.679410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.1.252.194:38083)
0928 15:01:46.679475 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.679476 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.679483 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:01:46.679510 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.679510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.679526 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.679610 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:46.679761 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.680273 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.681033 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.681235 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.681287 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.681415 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:46.681503 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:46.681584 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.681597 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.681612 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:01:46.683454 19485 raft_consensus.cc:1261] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 1 FOLLOWER]: Refusing update from remote peer 8a3d117d0422451ca158369566740a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:46.683589 19216 raft_consensus.cc:1261] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0 [term 1 FOLLOWER]: Refusing update from remote peer 8a3d117d0422451ca158369566740a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:46.683790 19556 consensus_queue.cc:1035] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "125a56f60a014f6c9b517f77c4f5de18" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:46.683948 19558 consensus_queue.cc:1035] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aaee8ee1a06479b831d6a0d0b04e7e0" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:46.686173 19561 mvcc.cc:205] Tried to move back new op lower bound from 7334329166572683264 to 7334329166532661248. Current Snapshot: MvccSnapshot[applied={T|T < 7334329166572683264}]
I20260928 15:01:46.687170 19563 mvcc.cc:205] Tried to move back new op lower bound from 7334329166572683264 to 7334329166532661248. Current Snapshot: MvccSnapshot[applied={T|T < 7334329166572683264}]
I20260928 15:01:46.688526 19070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:46.688657 19070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:46.689388 19070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8739d6942f44b67963f38c0b338857d: Sending DeleteTablet for 3 replicas of tablet 55d9c1fa17c74ad7b2d2ed370e415299
I20260928 15:01:46.689684 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d9c1fa17c74ad7b2d2ed370e415299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:46 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:34030
I20260928 15:01:46.689764 19465 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d9c1fa17c74ad7b2d2ed370e415299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:46 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59518
I20260928 15:01:46.689797 19196 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d9c1fa17c74ad7b2d2ed370e415299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:46 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:58852
I20260928 15:01:46.689999 19573 tablet_replica.cc:330] T 55d9c1fa17c74ad7b2d2ed370e415299 P 9aaee8ee1a06479b831d6a0d0b04e7e0: stopping tablet replica
I20260928 15:01:46.690071 19571 tablet_replica.cc:330] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: stopping tablet replica
I20260928 15:01:46.690136 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19132
I20260928 15:01:46.690164 19571 raft_consensus.cc:2229] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:46.691061 19571 raft_consensus.cc:2258] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:46.692278 19571 ts_tablet_manager.cc:1905] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:46.694013 19571 ts_tablet_manager.cc:1918] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:46.694090 19571 log.cc:1196] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/55d9c1fa17c74ad7b2d2ed370e415299
I20260928 15:01:46.694382 19571 ts_tablet_manager.cc:1939] T 55d9c1fa17c74ad7b2d2ed370e415299 P 8a3d117d0422451ca158369566740a3a: Deleting consensus metadata
I20260928 15:01:46.694813 19055 catalog_manager.cc:4841] TS 8a3d117d0422451ca158369566740a3a (127.1.252.194:38083): tablet 55d9c1fa17c74ad7b2d2ed370e415299 (table test-table [id=76578218a83447e6b444e59955d27bf5]) successfully deleted
W20260928 15:01:46.696183 19057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:38507 (error 108)
I20260928 15:01:46.696385 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19267
I20260928 15:01:46.696996 19572 tablet_replica.cc:330] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: stopping tablet replica
I20260928 15:01:46.697121 19572 raft_consensus.cc:2229] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:46.697229 19572 raft_consensus.cc:2258] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:46.698004 19572 ts_tablet_manager.cc:1905] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:46.698647 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.698501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:38507 (local address 127.0.0.1:58860)
0928 15:01:46.698560 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:46.698607 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:01:46.698767 19057 catalog_manager.cc:4584] TS 9aaee8ee1a06479b831d6a0d0b04e7e0 (127.1.252.193:38507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55d9c1fa17c74ad7b2d2ed370e415299: Network error: Client connection negotiation failed: client connection to 127.1.252.193:38507: connect: Connection refused (error 111)
I20260928 15:01:46.699716 19572 ts_tablet_manager.cc:1918] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:46.699772 19572 log.cc:1196] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/55d9c1fa17c74ad7b2d2ed370e415299
I20260928 15:01:46.700083 19572 ts_tablet_manager.cc:1939] T 55d9c1fa17c74ad7b2d2ed370e415299 P 125a56f60a014f6c9b517f77c4f5de18: Deleting consensus metadata
I20260928 15:01:46.700474 19057 catalog_manager.cc:4841] TS 125a56f60a014f6c9b517f77c4f5de18 (127.1.252.195:41295): tablet 55d9c1fa17c74ad7b2d2ed370e415299 (table test-table [id=76578218a83447e6b444e59955d27bf5]) successfully deleted
I20260928 15:01:46.702111 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19402
I20260928 15:01:46.706796 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19037
2026-09-28T15:01:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1955 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T15:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:46Z Disabled control of system clock
I20260928 15:01:46.734172 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38691
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:38691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:46.815641 19578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.815862 19578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.815944 19578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.817070 19578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:46.817111 19578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.817124 19578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.817137 19578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:46.817148 19578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:46.818439 19578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:38691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:46.818639 19578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.818872 19578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:46.823792 19586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.823805 19584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.823982 19583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.824005 19578 server_base.cc:1034] running on GCE node
I20260928 15:01:46.824255 19578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.824505 19578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.825670 19578 hybrid_clock.cc:648] HybridClock initialized: now 1790607706825653 us; error 28 us; skew 500 ppm
I20260928 15:01:46.827136 19578 webserver.cc:460] Webserver started at http://127.1.252.254:42795/ using document root <none> and password file <none>
I20260928 15:01:46.827358 19578 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.827405 19578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.827497 19578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.828609 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "660696468e4243af8f97750f15f34f7b"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "d54937936480909c646872ae9932616c"
server_key_iv: "aee159db1b927ce399a6ed4f10494cbf"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.828939 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "660696468e4243af8f97750f15f34f7b"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "d54937936480909c646872ae9932616c"
server_key_iv: "aee159db1b927ce399a6ed4f10494cbf"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.830319 19578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:01:46.831063 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.831259 19578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:46.831333 19578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "660696468e4243af8f97750f15f34f7b"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "d54937936480909c646872ae9932616c"
server_key_iv: "aee159db1b927ce399a6ed4f10494cbf"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.831388 19578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:46.844310 19578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:46.844585 19578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:46.844688 19578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:46.848819 19578 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:38691
I20260928 15:01:46.848884 19644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:38691 every 8 connection(s)
I20260928 15:01:46.849206 19578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:46.850152 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:46.852413 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b: Bootstrap starting.
I20260928 15:01:46.852934 19645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:46.853171 19645 log.cc:824] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:46.853868 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b: No bootstrap required, opened a new log
I20260928 15:01:46.855764 19645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660696468e4243af8f97750f15f34f7b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38691 } }
I20260928 15:01:46.856022 19645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:46.856081 19645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660696468e4243af8f97750f15f34f7b, State: Initialized, Role: FOLLOWER
I20260928 15:01:46.856247 19645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660696468e4243af8f97750f15f34f7b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38691 } }
I20260928 15:01:46.856346 19645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:46.856395 19645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:46.856446 19645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:46.857018 19645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660696468e4243af8f97750f15f34f7b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38691 } }
I20260928 15:01:46.857147 19645 leader_election.cc:302] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 660696468e4243af8f97750f15f34f7b; no voters:
I20260928 15:01:46.857336 19645 leader_election.cc:288] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:46.857383 19648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:46.857605 19648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [term 1 LEADER]: Becoming Leader. State: Replica: 660696468e4243af8f97750f15f34f7b, State: Running, Role: LEADER
I20260928 15:01:46.857645 19645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:46.857725 19648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660696468e4243af8f97750f15f34f7b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38691 } }
I20260928 15:01:46.858155 19650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 660696468e4243af8f97750f15f34f7b. Latest consensus state: current_term: 1 leader_uuid: "660696468e4243af8f97750f15f34f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660696468e4243af8f97750f15f34f7b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38691 } } }
I20260928 15:01:46.858260 19650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:46.858512 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19578
I20260928 15:01:46.858615 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:46.858237 19649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "660696468e4243af8f97750f15f34f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660696468e4243af8f97750f15f34f7b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38691 } } }
I20260928 15:01:46.858829 2035 external_mini_cluster.cc:1428] Setting key ff631db94eaabab64e425884b3184b46
I20260928 15:01:46.858914 19649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:46.859164 19657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:46.859707 19657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:46.862152 19657 catalog_manager.cc:1347] Generated new cluster ID: 908547c34de04fdca64315c3315cfc97
I20260928 15:01:46.862208 19657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:46.864591 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:46.859324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.1.252.254:38691)
0928 15:01:46.859543 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:46.859554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:46.859589 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:01:46.859708 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:46.859715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:46.859784 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:46.860199 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:46.860290 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.861610 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.862564 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:46.862986 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:46.863125 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:46.863858 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:46.863862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:46.863862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:46.864101 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:46.864133 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:46.864145 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:46.864225 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:46.864301 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:01:46.864378 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 15:01:46.882504 19657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:46.883064 19657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:46.889142 19657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b: Generated new TSK 0
I20260928 15:01:46.889333 19657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:46.894272 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38691
--builtin_ntp_servers=127.1.252.212:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:46.978366 19669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:46.978613 19669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:46.978649 19669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:46.979794 19669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:46.979858 19669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:46.979918 19669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:46.981261 19669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:46.981523 19669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:46.981818 19669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:01:46.987898 19669 server_base.cc:1034] running on GCE node
W20260928 15:01:46.987911 19675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.988014 19677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:46.988178 19674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:46.988463 19669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:46.988735 19669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:46.989931 19669 hybrid_clock.cc:648] HybridClock initialized: now 1790607706989914 us; error 33 us; skew 500 ppm
I20260928 15:01:46.991276 19669 webserver.cc:460] Webserver started at http://127.1.252.193:45733/ using document root <none> and password file <none>
I20260928 15:01:46.991528 19669 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:46.991575 19669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:46.991709 19669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:46.992856 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "89f701302ca445a29b87932a8e4223fb"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "da2888fb1ee4790042b7e5b9b7c1bf5f"
server_key_iv: "be08042491928775f236a9d2585b671d"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.993221 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "89f701302ca445a29b87932a8e4223fb"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "da2888fb1ee4790042b7e5b9b7c1bf5f"
server_key_iv: "be08042491928775f236a9d2585b671d"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.994680 19669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:01:46.995568 19683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:46.995760 19669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:46.995859 19669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "89f701302ca445a29b87932a8e4223fb"
format_stamp: "Formatted at 2026-09-28 15:01:46 on dist-test-slave-sf70"
server_key: "da2888fb1ee4790042b7e5b9b7c1bf5f"
server_key_iv: "be08042491928775f236a9d2585b671d"
server_key_version: "encryptionkey@0"
I20260928 15:01:46.995954 19669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:47.022306 19669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:47.022672 19669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:47.022832 19669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:47.023080 19669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:47.023425 19669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:47.023483 19669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.023526 19669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:47.023566 19669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.029863 19669 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:46865
I20260928 15:01:47.029923 19797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:46865 every 8 connection(s)
I20260928 15:01:47.030284 19669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:47.035764 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.031058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:47801 (local address 127.1.252.254:38691)
0928 15:01:47.031132 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.031135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.031385 (+ 250us) server_negotiation.cc:407] Connection header received
0928 15:01:47.031869 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.031870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.031930 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.032004 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:47.032718 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.033246 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.034408 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.034607 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.034663 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.035505 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:47.035506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:47.035506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:47.035512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:47.035527 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:47.035531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:47.035585 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.035671 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.035702 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:01:47.035967 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.031031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38691 (local address 127.1.252.193:47801)
0928 15:01:47.031260 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.031295 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.031786 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.032073 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.032076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.032084 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:47.032623 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.032629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.033329 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.033329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.034331 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.034334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.034691 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.034691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.034799 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.035314 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:47.035314 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:47.035342 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:47.035365 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:47.035597 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:47.035602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:47.035604 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.035657 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.035738 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260928 15:01:47.036509 19798 heartbeater.cc:347] Connected to a master server at 127.1.252.254:38691
I20260928 15:01:47.036648 19798 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:47.036828 19798 heartbeater.cc:511] Master 127.1.252.254:38691 requested a full tablet report, sending...
I20260928 15:01:47.037343 19609 ts_manager.cc:194] Registered new tserver with Master: 89f701302ca445a29b87932a8e4223fb (127.1.252.193:46865)
I20260928 15:01:47.038002 19609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.193:47801
I20260928 15:01:47.039769 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19669
I20260928 15:01:47.039880 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:47.040124 2035 external_mini_cluster.cc:1428] Setting key f002a2d134ce532a689dcf939deb9575
I20260928 15:01:47.041010 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38691
--builtin_ntp_servers=127.1.252.212:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:47.122818 19801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:47.123071 19801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:47.123095 19801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:47.124313 19801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:47.124356 19801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:47.124382 19801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:47.125775 19801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:47.126025 19801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:47.126276 19801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:47.131345 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:47.131332 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:47.131345 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:47.131808 19801 server_base.cc:1034] running on GCE node
I20260928 15:01:47.132094 19801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:47.132340 19801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:47.133504 19801 hybrid_clock.cc:648] HybridClock initialized: now 1790607707133487 us; error 44 us; skew 500 ppm
I20260928 15:01:47.134833 19801 webserver.cc:460] Webserver started at http://127.1.252.194:40165/ using document root <none> and password file <none>
I20260928 15:01:47.135095 19801 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:47.135169 19801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:47.135318 19801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:47.136525 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "0144c89e2563482ebdbad972c5422219"
format_stamp: "Formatted at 2026-09-28 15:01:47 on dist-test-slave-sf70"
server_key: "aa86058784c37697c8acccc2f1fe7715"
server_key_iv: "ce6d700f46f3cf22df49d304575d445e"
server_key_version: "encryptionkey@0"
I20260928 15:01:47.136880 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "0144c89e2563482ebdbad972c5422219"
format_stamp: "Formatted at 2026-09-28 15:01:47 on dist-test-slave-sf70"
server_key: "aa86058784c37697c8acccc2f1fe7715"
server_key_iv: "ce6d700f46f3cf22df49d304575d445e"
server_key_version: "encryptionkey@0"
I20260928 15:01:47.138161 19801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:01:47.138960 19816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.139168 19801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:47.139266 19801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "0144c89e2563482ebdbad972c5422219"
format_stamp: "Formatted at 2026-09-28 15:01:47 on dist-test-slave-sf70"
server_key: "aa86058784c37697c8acccc2f1fe7715"
server_key_iv: "ce6d700f46f3cf22df49d304575d445e"
server_key_version: "encryptionkey@0"
I20260928 15:01:47.139348 19801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:47.162765 19801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:47.163103 19801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:47.163269 19801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:47.163514 19801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:47.163873 19801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:47.163980 19801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.164026 19801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:47.164055 19801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.170019 19801 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:42127
I20260928 15:01:47.170073 19929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:42127 every 8 connection(s)
I20260928 15:01:47.170395 19801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:47.176240 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.171157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:43953 (local address 127.1.252.254:38691)
0928 15:01:47.171227 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.171231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.171501 (+ 270us) server_negotiation.cc:407] Connection header received
0928 15:01:47.172102 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.172104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.172118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.172195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:47.173250 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.173756 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.174744 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.174941 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.175008 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.175835 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:47.175836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:47.175836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:47.175841 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:47.175854 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:47.175858 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:47.175968 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.176109 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.176142 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:01:47.176271 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19801
I20260928 15:01:47.176359 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:47.176450 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.171160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38691 (local address 127.1.252.194:43953)
0928 15:01:47.171374 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.171410 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.171941 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.172279 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.172284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.172295 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:47.173115 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.173121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.173846 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.173847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.174651 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.174653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.175062 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.175063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.175182 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.175697 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:47.175698 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:47.175727 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:47.175749 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:47.175979 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:47.175986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:47.175989 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.176087 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.176175 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260928 15:01:47.176605 2035 external_mini_cluster.cc:1428] Setting key 80ac2fadaee95cbde286e6e8dbd45d3f
I20260928 15:01:47.177052 19930 heartbeater.cc:347] Connected to a master server at 127.1.252.254:38691
I20260928 15:01:47.177161 19930 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:47.177364 19930 heartbeater.cc:511] Master 127.1.252.254:38691 requested a full tablet report, sending...
I20260928 15:01:47.177686 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38691
--builtin_ntp_servers=127.1.252.212:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:47.177799 19609 ts_manager.cc:194] Registered new tserver with Master: 0144c89e2563482ebdbad972c5422219 (127.1.252.194:42127)
I20260928 15:01:47.178154 19609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.194:43953
W20260928 15:01:47.260653 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:47.260852 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:47.260874 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:47.262037 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:47.262080 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:47.262122 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:47.263468 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:38691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:47.263715 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:47.264076 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:47.269420 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:47.269428 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:47.269443 19933 server_base.cc:1034] running on GCE node
W20260928 15:01:47.269490 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:47.269887 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:47.270179 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:47.271347 19933 hybrid_clock.cc:648] HybridClock initialized: now 1790607707271328 us; error 40 us; skew 500 ppm
I20260928 15:01:47.272733 19933 webserver.cc:460] Webserver started at http://127.1.252.195:42945/ using document root <none> and password file <none>
I20260928 15:01:47.272986 19933 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:47.273059 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:47.273219 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:47.274277 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "647a4299eedc4b9c9fb51ceac9cc9d28"
format_stamp: "Formatted at 2026-09-28 15:01:47 on dist-test-slave-sf70"
server_key: "a2f3837c7e0554f71fb0df526c8adc78"
server_key_iv: "1cd0b523a3035aba79f17962a1848cff"
server_key_version: "encryptionkey@0"
I20260928 15:01:47.274644 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "647a4299eedc4b9c9fb51ceac9cc9d28"
format_stamp: "Formatted at 2026-09-28 15:01:47 on dist-test-slave-sf70"
server_key: "a2f3837c7e0554f71fb0df526c8adc78"
server_key_iv: "1cd0b523a3035aba79f17962a1848cff"
server_key_version: "encryptionkey@0"
I20260928 15:01:47.276017 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:47.276767 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.276952 19933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:47.277048 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "647a4299eedc4b9c9fb51ceac9cc9d28"
format_stamp: "Formatted at 2026-09-28 15:01:47 on dist-test-slave-sf70"
server_key: "a2f3837c7e0554f71fb0df526c8adc78"
server_key_iv: "1cd0b523a3035aba79f17962a1848cff"
server_key_version: "encryptionkey@0"
I20260928 15:01:47.277144 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:47.286564 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:47.286860 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:47.287014 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:47.287226 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:47.287547 19933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:47.287603 19933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.287640 19933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:47.287668 19933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:47.293495 19933 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:41965
I20260928 15:01:47.293541 20061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:41965 every 8 connection(s)
I20260928 15:01:47.293860 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:47.299031 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.294565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:36815 (local address 127.1.252.254:38691)
0928 15:01:47.294647 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.294649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.294852 (+ 203us) server_negotiation.cc:407] Connection header received
0928 15:01:47.295325 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.295326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.295336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.295389 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:47.296111 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.296604 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.297577 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.297778 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.297854 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.298804 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:47.298806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:47.298806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:47.298811 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:47.298825 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:47.298829 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:47.298879 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.298946 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.298971 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:01:47.299288 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.294536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38691 (local address 127.1.252.195:36815)
0928 15:01:47.294736 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.294769 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.295244 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.295454 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.295457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.295465 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:47.296034 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.296039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.296701 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.296702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.297482 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.297484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.297876 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.297877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.298056 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.298656 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:47.298656 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:47.298686 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:47.298717 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:47.298890 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:47.298894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:47.298896 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.298949 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.299031 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260928 15:01:47.299815 20062 heartbeater.cc:347] Connected to a master server at 127.1.252.254:38691
I20260928 15:01:47.299957 20062 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:47.300136 20062 heartbeater.cc:511] Master 127.1.252.254:38691 requested a full tablet report, sending...
I20260928 15:01:47.300518 19609 ts_manager.cc:194] Registered new tserver with Master: 647a4299eedc4b9c9fb51ceac9cc9d28 (127.1.252.195:41965)
I20260928 15:01:47.300860 19609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.195:36815
I20260928 15:01:47.302738 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 19933
I20260928 15:01:47.302821 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:47.302989 2035 external_mini_cluster.cc:1428] Setting key 88d9a956542f7edd359af57846a0f652
I20260928 15:01:47.304033 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:47.308353 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.305367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.1.252.254:38691)
0928 15:01:47.305446 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.305447 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.305616 (+ 169us) server_negotiation.cc:407] Connection header received
0928 15:01:47.305636 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.305637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.305648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.305696 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:47.305876 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.306346 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.307163 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.307333 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.307381 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.308083 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:47.308085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:47.308085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:47.308089 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:47.308102 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:47.308106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:47.308150 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.308218 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.308240 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:01:47.310002 19609 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:47.311594 19609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:47.318223 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.314220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:42127 (local address 127.0.0.1:54276)
0928 15:01:47.314822 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.314895 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.314944 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.315197 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.315198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.315200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:47.315234 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.315235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.316375 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.316376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.317255 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.317257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.317941 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.317942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.318015 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.318028 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.318103 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.318152 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":70,"threads_started":1}
I20260928 15:01:47.318235 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.314462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.1.252.194:42127)
0928 15:01:47.314644 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.314649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.314999 (+ 350us) server_negotiation.cc:407] Connection header received
0928 15:01:47.315052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.315054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.315068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.315135 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:47.315312 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.316289 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.317352 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.317846 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.317941 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.318031 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.318116 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.318151 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260928 15:01:47.318476 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.314009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:46865 (local address 127.0.0.1:37042)
0928 15:01:47.314121 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.314161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.314220 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.314723 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.314724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.314727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:47.314769 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.314773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.315846 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.315847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.317296 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.317299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.318214 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.318215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.318278 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.318282 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.318313 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.318366 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":70,"thread_start_us":68,"threads_started":1}
I20260928 15:01:47.318831 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.314148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.1.252.193:46865)
0928 15:01:47.314569 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.314573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.314584 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:01:47.314626 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.314628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.314644 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.314725 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:47.314856 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.315756 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.317399 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.318107 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.318600 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.318693 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.318717 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.318754 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":86,"threads_started":1}
I20260928 15:01:47.319232 19864 tablet_service.cc:1467] Processing CreateTablet for tablet 66a0265055944466813b1f210b618e4a (DEFAULT_TABLE table=test-table [id=c715fb7cf60d4aa7a8fa149983ae9171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:47.319585 19864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a0265055944466813b1f210b618e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:47.319566 19732 tablet_service.cc:1467] Processing CreateTablet for tablet 66a0265055944466813b1f210b618e4a (DEFAULT_TABLE table=test-table [id=c715fb7cf60d4aa7a8fa149983ae9171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:47.319947 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a0265055944466813b1f210b618e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:47.322255 20077 tablet_bootstrap.cc:492] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Bootstrap starting.
I20260928 15:01:47.322676 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.314120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:41965 (local address 127.0.0.1:56004)
0928 15:01:47.314326 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.314338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.314414 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.314680 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.314683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.314685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:47.314748 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.314750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.316024 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.316025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.317296 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.317299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.322385 (+ 5086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.322386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.322467 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.322471 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.322526 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.322579 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":17,"thread_start_us":96,"threads_started":1}
I20260928 15:01:47.322708 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.314345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.1.252.195:41965)
0928 15:01:47.314489 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.314494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.314504 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:01:47.314546 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.314547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.314580 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.314666 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:47.314829 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.315945 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.317399 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.318128 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.318188 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.319937 (+ 1749us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.322562 (+ 2625us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.322614 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260928 15:01:47.322952 20078 tablet_bootstrap.cc:492] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Bootstrap starting.
I20260928 15:01:47.323182 20077 tablet_bootstrap.cc:654] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:47.323297 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 66a0265055944466813b1f210b618e4a (DEFAULT_TABLE table=test-table [id=c715fb7cf60d4aa7a8fa149983ae9171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:47.323541 20077 log.cc:824] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:47.323619 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a0265055944466813b1f210b618e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:47.323632 20078 tablet_bootstrap.cc:654] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:47.323993 20078 log.cc:824] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:47.324360 20077 tablet_bootstrap.cc:492] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: No bootstrap required, opened a new log
I20260928 15:01:47.324452 20077 ts_tablet_manager.cc:1397] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:47.324782 20078 tablet_bootstrap.cc:492] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: No bootstrap required, opened a new log
I20260928 15:01:47.324851 20078 ts_tablet_manager.cc:1397] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:01:47.326462 20081 tablet_bootstrap.cc:492] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Bootstrap starting.
I20260928 15:01:47.327076 20081 tablet_bootstrap.cc:654] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:47.327178 20077 raft_consensus.cc:348] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.327374 20077 raft_consensus.cc:374] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:47.327391 20081 log.cc:824] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:47.327419 20077 raft_consensus.cc:729] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0144c89e2563482ebdbad972c5422219, State: Initialized, Role: FOLLOWER
I20260928 15:01:47.327567 20077 consensus_queue.cc:260] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.327564 20078 raft_consensus.cc:348] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.327787 20078 raft_consensus.cc:374] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:47.327827 20078 raft_consensus.cc:729] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f701302ca445a29b87932a8e4223fb, State: Initialized, Role: FOLLOWER
I20260928 15:01:47.327837 20077 ts_tablet_manager.cc:1428] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 15:01:47.327988 19930 heartbeater.cc:503] Master 127.1.252.254:38691 was elected leader, sending a full tablet report...
I20260928 15:01:47.327986 20078 consensus_queue.cc:260] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.328143 20081 tablet_bootstrap.cc:492] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: No bootstrap required, opened a new log
I20260928 15:01:47.328205 20081 ts_tablet_manager.cc:1397] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:47.328228 20078 ts_tablet_manager.cc:1428] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 15:01:47.330338 20081 raft_consensus.cc:348] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.330464 20081 raft_consensus.cc:374] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:47.330487 20081 raft_consensus.cc:729] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647a4299eedc4b9c9fb51ceac9cc9d28, State: Initialized, Role: FOLLOWER
I20260928 15:01:47.330590 20081 consensus_queue.cc:260] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.330785 20081 ts_tablet_manager.cc:1428] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:01:47.330900 20062 heartbeater.cc:503] Master 127.1.252.254:38691 was elected leader, sending a full tablet report...
I20260928 15:01:47.330901 19798 heartbeater.cc:503] Master 127.1.252.254:38691 was elected leader, sending a full tablet report...
W20260928 15:01:47.421676 19931 tablet.cc:2369] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:47.490543 20083 raft_consensus.cc:482] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:47.490712 20083 raft_consensus.cc:504] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.491132 20083 leader_election.cc:288] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f701302ca445a29b87932a8e4223fb (127.1.252.193:46865), 647a4299eedc4b9c9fb51ceac9cc9d28 (127.1.252.195:41965)
I20260928 15:01:47.494093 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.491308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:41965 (local address 127.1.252.194:48191)
0928 15:01:47.491362 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.491374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.491452 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.491613 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.491614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.491617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:47.491714 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.491716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.492520 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.492521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.493273 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.493275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.493826 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.493827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.493900 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.493903 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.493972 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.494023 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 15:01:47.494094 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.491397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:48191 (local address 127.1.252.195:41965)
0928 15:01:47.491467 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.491470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.491512 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:01:47.491551 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.491552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.491560 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.491602 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:47.491811 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.492422 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.493366 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.493743 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.493828 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.493835 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.493990 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.494027 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 15:01:47.494309 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.491310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:46865 (local address 127.1.252.194:44919)
0928 15:01:47.491692 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.491702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:47.491757 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:47.491953 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:47.491953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:47.491955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:47.492040 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.492042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.492718 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.492719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.493470 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:47.493472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.494017 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.494018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:47.494090 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.494092 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:47.494159 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 15:01:47.494207 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":201,"thread_start_us":104,"threads_started":1}
I20260928 15:01:47.494401 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.491441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:44919 (local address 127.1.252.193:46865)
0928 15:01:47.491543 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.491545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.491770 (+ 225us) server_negotiation.cc:407] Connection header received
0928 15:01:47.491833 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.491838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.491855 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.491946 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:47.492118 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.492631 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.493565 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.493918 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.493998 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.494007 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.494281 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.494326 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:01:47.494623 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0265055944466813b1f210b618e4a" candidate_uuid: "0144c89e2563482ebdbad972c5422219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" is_pre_election: true
I20260928 15:01:47.494664 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0265055944466813b1f210b618e4a" candidate_uuid: "0144c89e2563482ebdbad972c5422219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f701302ca445a29b87932a8e4223fb" is_pre_election: true
I20260928 15:01:47.494798 19752 raft_consensus.cc:2454] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0144c89e2563482ebdbad972c5422219 in term 0.
I20260928 15:01:47.494798 20016 raft_consensus.cc:2454] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0144c89e2563482ebdbad972c5422219 in term 0.
I20260928 15:01:47.495033 19817 leader_election.cc:302] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0144c89e2563482ebdbad972c5422219, 89f701302ca445a29b87932a8e4223fb; no voters:
I20260928 15:01:47.495184 20083 raft_consensus.cc:2781] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:47.495240 20083 raft_consensus.cc:482] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:47.495266 20083 raft_consensus.cc:3037] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:47.496111 20083 raft_consensus.cc:504] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.496258 20083 leader_election.cc:288] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [CANDIDATE]: Term 1 election: Requested vote from peers 89f701302ca445a29b87932a8e4223fb (127.1.252.193:46865), 647a4299eedc4b9c9fb51ceac9cc9d28 (127.1.252.195:41965)
I20260928 15:01:47.496381 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0265055944466813b1f210b618e4a" candidate_uuid: "0144c89e2563482ebdbad972c5422219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f701302ca445a29b87932a8e4223fb"
I20260928 15:01:47.496416 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0265055944466813b1f210b618e4a" candidate_uuid: "0144c89e2563482ebdbad972c5422219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28"
I20260928 15:01:47.496481 19752 raft_consensus.cc:3037] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:47.496516 20016 raft_consensus.cc:3037] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:47.497197 19752 raft_consensus.cc:2454] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0144c89e2563482ebdbad972c5422219 in term 1.
I20260928 15:01:47.497252 20016 raft_consensus.cc:2454] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0144c89e2563482ebdbad972c5422219 in term 1.
I20260928 15:01:47.497412 19817 leader_election.cc:302] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0144c89e2563482ebdbad972c5422219, 89f701302ca445a29b87932a8e4223fb; no voters:
I20260928 15:01:47.497579 20083 raft_consensus.cc:2781] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:47.497772 20083 raft_consensus.cc:686] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 1 LEADER]: Becoming Leader. State: Replica: 0144c89e2563482ebdbad972c5422219, State: Running, Role: LEADER
I20260928 15:01:47.497866 20083 consensus_queue.cc:237] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } }
I20260928 15:01:47.498809 19608 catalog_manager.cc:5495] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0144c89e2563482ebdbad972c5422219 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "0144c89e2563482ebdbad972c5422219" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0144c89e2563482ebdbad972c5422219" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 42127 } health_report { overall_health: HEALTHY } } }
W20260928 15:01:47.531018 19799 tablet.cc:2369] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:47.544399 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.541932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.1.252.194:42127)
0928 15:01:47.541995 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:47.541997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:47.542006 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:01:47.542056 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:47.542057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:47.542075 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:47.542178 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:47.542324 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.542845 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.543588 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:47.543775 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:47.543849 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:47.544061 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:47.544158 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:47.544251 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:47.544269 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 15:01:47.544286 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 15:01:47.544384 20063 tablet.cc:2369] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:47.546231 19752 raft_consensus.cc:1261] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 1 FOLLOWER]: Refusing update from remote peer 0144c89e2563482ebdbad972c5422219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:47.546238 20016 raft_consensus.cc:1261] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 1 FOLLOWER]: Refusing update from remote peer 0144c89e2563482ebdbad972c5422219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:47.546593 20087 consensus_queue.cc:1035] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f701302ca445a29b87932a8e4223fb" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:47.546748 20083 consensus_queue.cc:1035] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "647a4299eedc4b9c9fb51ceac9cc9d28" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:47.549263 20091 mvcc.cc:205] Tried to move back new op lower bound from 7334329170106605568 to 7334329169911840768. Current Snapshot: MvccSnapshot[applied={T|T < 7334329170106605568}]
I20260928 15:01:47.551785 19608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:47.551908 19608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:47.552759 19608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 660696468e4243af8f97750f15f34f7b: Sending DeleteTablet for 3 replicas of tablet 66a0265055944466813b1f210b618e4a
I20260928 15:01:47.553133 19864 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a0265055944466813b1f210b618e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:47 UTC) from {username='slave'} at 127.0.0.1:54276
I20260928 15:01:47.553231 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a0265055944466813b1f210b618e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:47 UTC) from {username='slave'} at 127.0.0.1:37042
I20260928 15:01:47.553388 20100 tablet_replica.cc:330] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: stopping tablet replica
I20260928 15:01:47.553467 20101 tablet_replica.cc:330] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: stopping tablet replica
I20260928 15:01:47.553483 20100 raft_consensus.cc:2229] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:47.553570 20101 raft_consensus.cc:2229] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:47.553609 20100 raft_consensus.cc:2258] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:47.553603 19996 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a0265055944466813b1f210b618e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:47 UTC) from {username='slave'} at 127.0.0.1:56004
I20260928 15:01:47.553669 20101 raft_consensus.cc:2258] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:47.554060 20100 ts_tablet_manager.cc:1905] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:47.554140 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19669
I20260928 15:01:47.554147 20101 ts_tablet_manager.cc:1905] T 66a0265055944466813b1f210b618e4a P 89f701302ca445a29b87932a8e4223fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:47.555804 20100 ts_tablet_manager.cc:1918] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:47.555867 20100 log.cc:1196] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/66a0265055944466813b1f210b618e4a
I20260928 15:01:47.556166 20100 ts_tablet_manager.cc:1939] T 66a0265055944466813b1f210b618e4a P 0144c89e2563482ebdbad972c5422219: Deleting consensus metadata
I20260928 15:01:47.557346 19593 catalog_manager.cc:4841] TS 0144c89e2563482ebdbad972c5422219 (127.1.252.194:42127): tablet 66a0265055944466813b1f210b618e4a (table test-table [id=c715fb7cf60d4aa7a8fa149983ae9171]) successfully deleted
W20260928 15:01:47.560271 19593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:46865 (error 108)
I20260928 15:01:47.560828 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:47.560678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:46865 (local address 127.0.0.1:37044)
0928 15:01:47.560743 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:47.560789 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:46865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:01:47.560875 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19801
I20260928 15:01:47.560886 20102 tablet_replica.cc:330] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: stopping tablet replica
I20260928 15:01:47.561017 20102 raft_consensus.cc:2229] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:47.561128 20102 raft_consensus.cc:2258] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 15:01:47.562281 19593 catalog_manager.cc:4584] TS 89f701302ca445a29b87932a8e4223fb (127.1.252.193:46865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66a0265055944466813b1f210b618e4a: Network error: Client connection negotiation failed: client connection to 127.1.252.193:46865: connect: Connection refused (error 111)
I20260928 15:01:47.562477 20102 ts_tablet_manager.cc:1905] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:47.564033 20102 ts_tablet_manager.cc:1918] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:47.564113 20102 log.cc:1196] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/66a0265055944466813b1f210b618e4a
I20260928 15:01:47.564361 20102 ts_tablet_manager.cc:1939] T 66a0265055944466813b1f210b618e4a P 647a4299eedc4b9c9fb51ceac9cc9d28: Deleting consensus metadata
I20260928 15:01:47.564801 19594 catalog_manager.cc:4841] TS 647a4299eedc4b9c9fb51ceac9cc9d28 (127.1.252.195:41965): tablet 66a0265055944466813b1f210b618e4a (table test-table [id=c715fb7cf60d4aa7a8fa149983ae9171]) successfully deleted
I20260928 15:01:47.566555 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19933
I20260928 15:01:47.570950 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 19578
2026-09-28T15:01:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (863 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:47 dist-test-slave-sf70 krb5kdc[20106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:47 dist-test-slave-sf70 krb5kdc[20106](info): set up 2 sockets
Sep 28 15:01:47 dist-test-slave-sf70 krb5kdc[20106](info): commencing operation
krb5kdc: starting...
W20260928 15:01:48.638013 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:48 dist-test-slave-sf70 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:01:48.759835 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35671
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:48.841310 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:48.841543 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:48.841579 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:48.842746 20119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:48.842813 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:48.842836 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:48.842857 20119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:48.842888 20119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:48.844480 20119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:48.844738 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:48.844959 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:01:48.850059 20119 server_base.cc:1034] running on GCE node
W20260928 15:01:48.849951 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:48.849949 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:48.849997 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:48.850493 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:48.850752 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:48.851951 20119 hybrid_clock.cc:648] HybridClock initialized: now 1790607708851928 us; error 43 us; skew 500 ppm
Sep 28 15:01:48 dist-test-slave-sf70 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607708, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:48.854386 20119 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:01:48.854830 20119 webserver.cc:460] Webserver started at http://127.1.252.254:35957/ using document root <none> and password file <none>
I20260928 15:01:48.855033 20119 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:48.855073 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:48.855160 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:48.856247 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "86e84404073b4bcfa73dac8a2be6d5bb"
format_stamp: "Formatted at 2026-09-28 15:01:48 on dist-test-slave-sf70"
server_key: "9637898df518d6aa08397a1312c0a17f"
server_key_iv: "029a852457725a9869cddd776423bdbf"
server_key_version: "encryptionkey@0"
I20260928 15:01:48.856542 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "86e84404073b4bcfa73dac8a2be6d5bb"
format_stamp: "Formatted at 2026-09-28 15:01:48 on dist-test-slave-sf70"
server_key: "9637898df518d6aa08397a1312c0a17f"
server_key_iv: "029a852457725a9869cddd776423bdbf"
server_key_version: "encryptionkey@0"
I20260928 15:01:48.858161 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:01:48.858889 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:48.859069 20119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:48.859169 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "86e84404073b4bcfa73dac8a2be6d5bb"
format_stamp: "Formatted at 2026-09-28 15:01:48 on dist-test-slave-sf70"
server_key: "9637898df518d6aa08397a1312c0a17f"
server_key_iv: "029a852457725a9869cddd776423bdbf"
server_key_version: "encryptionkey@0"
I20260928 15:01:48.859262 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:48.888554 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:48.890588 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:48.890760 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:48.894901 20119 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:35671
I20260928 15:01:48.894945 20186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:35671 every 8 connection(s)
I20260928 15:01:48.895321 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:48.896294 20187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:48.898730 20187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb: Bootstrap starting.
I20260928 15:01:48.899344 20187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:48.899664 20187 log.cc:824] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:48.900529 20187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb: No bootstrap required, opened a new log
I20260928 15:01:48.902251 20187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35671 } }
I20260928 15:01:48.902387 20187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:48.902417 20187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e84404073b4bcfa73dac8a2be6d5bb, State: Initialized, Role: FOLLOWER
I20260928 15:01:48.902587 20187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35671 } }
I20260928 15:01:48.902650 20187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:48.902671 20187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:48.902742 20187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:48.903353 20187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35671 } }
I20260928 15:01:48.903450 20187 leader_election.cc:302] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e84404073b4bcfa73dac8a2be6d5bb; no voters:
I20260928 15:01:48.903636 20187 leader_election.cc:288] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:48.903798 20190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:48.903911 20187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:48.903996 20190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [term 1 LEADER]: Becoming Leader. State: Replica: 86e84404073b4bcfa73dac8a2be6d5bb, State: Running, Role: LEADER
I20260928 15:01:48.904088 20190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35671 } }
I20260928 15:01:48.904452 20192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e84404073b4bcfa73dac8a2be6d5bb. Latest consensus state: current_term: 1 leader_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35671 } } }
I20260928 15:01:48.904538 20192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:48.904690 20191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e84404073b4bcfa73dac8a2be6d5bb" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35671 } } }
I20260928 15:01:48.904757 20198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:48.904793 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20119
I20260928 15:01:48.904913 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:48.905136 2035 external_mini_cluster.cc:1428] Setting key bc1da3a7df32fc802213503938ea8b55
I20260928 15:01:48.905426 20198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:48.906222 20191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:48.907325 20198 catalog_manager.cc:1347] Generated new cluster ID: 78091b8675c240f7ac7a8d54a0cc0166
I20260928 15:01:48.907382 20198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:01:48 dist-test-slave-sf70 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:48.915470 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:48.905783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.1.252.254:35671)
0928 15:01:48.906364 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:48.906374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:48.906405 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:01:48.906525 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:48.906532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:48.906599 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:48.906729 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:48.907514 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:48.908515 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:48.909284 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:48.909748 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:48.909876 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:48.912235 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:48.912239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:48.912240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:48.912260 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:48.913931 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:48.914297 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:48.914300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:48.914301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:48.914407 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:48.914628 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:48.914628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:48.914629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:48.914871 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:48.915044 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:48.915118 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 15:01:48.915261 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":65,"threads_started":1}
I20260928 15:01:48.924710 20198 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:48.925532 20198 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:48.931480 20198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb: Generated new TSK 0
I20260928 15:01:48.931663 20198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 15:01:48.959785 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35671
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:49.039989 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:49.040205 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:49.040239 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:49.041383 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:49.041450 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:49.041491 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:49.042917 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:49.043182 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:49.043409 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:49.048751 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.048751 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.048779 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.049009 20213 server_base.cc:1034] running on GCE node
I20260928 15:01:49.049217 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:49.049450 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:49.050645 20213 hybrid_clock.cc:648] HybridClock initialized: now 1790607709050629 us; error 27 us; skew 500 ppm
Sep 28 15:01:49 dist-test-slave-sf70 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:49.053282 20213 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:01:49.053735 20213 webserver.cc:460] Webserver started at http://127.1.252.193:38501/ using document root <none> and password file <none>
I20260928 15:01:49.053960 20213 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:49.054028 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:49.054175 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:49.055253 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "00ca6004f9634ee7b76760ac970e81f5"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "9e2b7a483189ec97e507b8b18b48f803"
server_key_iv: "45cf2271dd06e44f03db77576c75b678"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.055624 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "00ca6004f9634ee7b76760ac970e81f5"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "9e2b7a483189ec97e507b8b18b48f803"
server_key_iv: "45cf2271dd06e44f03db77576c75b678"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.057046 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:01:49.057901 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.058135 20213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:49.058238 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "00ca6004f9634ee7b76760ac970e81f5"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "9e2b7a483189ec97e507b8b18b48f803"
server_key_iv: "45cf2271dd06e44f03db77576c75b678"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.058327 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:49.071717 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:49.073808 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:49.073978 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:49.074222 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:49.074542 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:49.074616 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.074658 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:49.074674 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.081100 20213 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:37789
I20260928 15:01:49.081188 20341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:37789 every 8 connection(s)
I20260928 15:01:49.081501 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:49.084882 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20213
I20260928 15:01:49.084978 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:49.085211 2035 external_mini_cluster.cc:1428] Setting key b40150621ba3c6bdcf2d929ba162d229
Sep 28 15:01:49 dist-test-slave-sf70 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:49.092346 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.082227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35671 (local address 127.1.252.193:57869)
0928 15:01:49.082404 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.082446 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.082970 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.083240 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.083243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.083813 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:49.084112 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.084117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.084870 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.084872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.086151 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.086154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.086542 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.086543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.086679 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.087217 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:49.087230 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:49.088638 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:49.090798 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:49.090806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:49.090807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:49.091065 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:49.091320 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:49.091321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:49.091321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:49.091441 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:49.091756 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:49.091761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:49.091937 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.092054 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.092182 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260928 15:01:49.092336 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.082277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:57869 (local address 127.1.252.254:35671)
0928 15:01:49.082324 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.082330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.082523 (+ 193us) server_negotiation.cc:407] Connection header received
0928 15:01:49.083072 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.083073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.083087 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.083157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:49.084210 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.084748 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.086239 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.086448 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.086532 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.088736 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:49.088738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:49.088738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:49.088746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:49.090686 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.091149 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.091150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.091151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.091220 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.091523 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.091524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.091524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.091676 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:49.091755 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.092103 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.092211 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:01:49.092927 20342 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35671
I20260928 15:01:49.093034 20342 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:49.093230 20342 heartbeater.cc:511] Master 127.1.252.254:35671 requested a full tablet report, sending...
I20260928 15:01:49.093822 20151 ts_manager.cc:194] Registered new tserver with Master: 00ca6004f9634ee7b76760ac970e81f5 (127.1.252.193:37789)
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 15:01:49.094532 20151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:57869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 15:01:49.111275 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35671
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:49.192178 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:49.192407 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:49.192448 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:49.193578 20347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:49.193619 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:49.193681 20347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:49.195545 20347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:49.195827 20347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:49.196115 20347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:49.201783 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.201803 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.201838 20347 server_base.cc:1034] running on GCE node
W20260928 15:01:49.201803 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.202227 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:49.202473 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:49.203617 20347 hybrid_clock.cc:648] HybridClock initialized: now 1790607709203618 us; error 41 us; skew 500 ppm
Sep 28 15:01:49 dist-test-slave-sf70 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:49.206660 20347 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:01:49.207126 20347 webserver.cc:460] Webserver started at http://127.1.252.194:39343/ using document root <none> and password file <none>
I20260928 15:01:49.207362 20347 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:49.207407 20347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:49.207566 20347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:49.208685 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "d125b0291de34d3d93458d04d9b1e6bf"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "aae4e8f2d27068acaabc0f24157d6119"
server_key_iv: "e3b150bddef86c2934dd2d0f5312e12b"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.209029 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "d125b0291de34d3d93458d04d9b1e6bf"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "aae4e8f2d27068acaabc0f24157d6119"
server_key_iv: "e3b150bddef86c2934dd2d0f5312e12b"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.210430 20347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:01:49.211179 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.211341 20347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 15:01:49.211431 20347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "d125b0291de34d3d93458d04d9b1e6bf"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "aae4e8f2d27068acaabc0f24157d6119"
server_key_iv: "e3b150bddef86c2934dd2d0f5312e12b"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.211508 20347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:49.228811 20347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:49.231061 20347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:49.231208 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:49.231437 20347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:49.231739 20347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:49.231793 20347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.231837 20347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:49.231864 20347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.237509 20347 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:34147
I20260928 15:01:49.237587 20476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:34147 every 8 connection(s)
I20260928 15:01:49.237900 20347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
Sep 28 15:01:49 dist-test-slave-sf70 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:49.246065 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20347
I20260928 15:01:49.246194 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:49.246433 2035 external_mini_cluster.cc:1428] Setting key 80cec2d8f85a42868096250e3f574b33
I20260928 15:01:49.247303 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.238602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:36883 (local address 127.1.252.254:35671)
0928 15:01:49.238671 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.238673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.238899 (+ 226us) server_negotiation.cc:407] Connection header received
0928 15:01:49.239374 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.239375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.239385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.239436 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:49.240337 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.240840 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.241836 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.242072 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.242155 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.244400 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:49.244401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:49.244402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:49.244410 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:49.245788 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.246203 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.246205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.246205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.246274 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.246612 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.246613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.246614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.246765 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:49.246857 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.247132 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.247215 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:01:49.247514 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.238580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35671 (local address 127.1.252.194:36883)
0928 15:01:49.238752 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.238787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.239285 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.239497 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.239501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.240071 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:49.240256 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.240260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.240934 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.240935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.241736 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.241738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.242170 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.242171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.242277 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.242734 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:49.242742 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:49.244285 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:49.245868 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:49.245874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:49.245875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:49.246092 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:49.246373 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:49.246374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:49.246375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:49.246532 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:49.246849 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:49.246856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:49.246991 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.247099 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.247254 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 15:01:49.248075 20477 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35671
I20260928 15:01:49.248186 20477 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:49.248404 20477 heartbeater.cc:511] Master 127.1.252.254:35671 requested a full tablet report, sending...
I20260928 15:01:49.248829 20151 ts_manager.cc:194] Registered new tserver with Master: d125b0291de34d3d93458d04d9b1e6bf (127.1.252.194:34147)
I20260928 15:01:49.249315 20151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:36883
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 15:01:49.275004 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35671
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:49.355618 20482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:49.355835 20482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:49.355870 20482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:49.357008 20482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:49.357076 20482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:49.357123 20482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:49.358541 20482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:49.358785 20482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:49.359002 20482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:49.364199 20487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.364243 20482 server_base.cc:1034] running on GCE node
W20260928 15:01:49.364291 20488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.364199 20490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.364583 20482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:49.364831 20482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:49.366025 20482 hybrid_clock.cc:648] HybridClock initialized: now 1790607709366005 us; error 28 us; skew 500 ppm
Sep 28 15:01:49 dist-test-slave-sf70 krb5kdc[20106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:49.368780 20482 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:01:49.369207 20482 webserver.cc:460] Webserver started at http://127.1.252.195:37131/ using document root <none> and password file <none>
I20260928 15:01:49.369436 20482 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:49.369505 20482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:49.369668 20482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:49.370724 20482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "1a7addfeaa174045b4e4e3487cfd745c"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "6135e80c385f0f06c5c15a3fa98ecd2b"
server_key_iv: "c3a2258bf334030931bf67e42364fc35"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.371085 20482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "1a7addfeaa174045b4e4e3487cfd745c"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "6135e80c385f0f06c5c15a3fa98ecd2b"
server_key_iv: "c3a2258bf334030931bf67e42364fc35"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.372480 20482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:01:49.373299 20497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.373672 20482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:49.373775 20482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "1a7addfeaa174045b4e4e3487cfd745c"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "6135e80c385f0f06c5c15a3fa98ecd2b"
server_key_iv: "c3a2258bf334030931bf67e42364fc35"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.373848 20482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:49.398247 20482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:49.400410 20482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:49.400597 20482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:49.400843 20482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:49.401152 20482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:49.401209 20482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.401250 20482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:49.401297 20482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.407829 20482 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:40957
I20260928 15:01:49.407872 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:40957 every 8 connection(s)
I20260928 15:01:49.408275 20482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:49.410038 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20482
I20260928 15:01:49.410126 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:49.410303 2035 external_mini_cluster.cc:1428] Setting key 4b1fc2261275252cefeb701583a4e701
Sep 28 15:01:49 dist-test-slave-sf70 krb5kdc[20106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607709, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:49.417824 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.409003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:46903 (local address 127.1.252.254:35671)
0928 15:01:49.409099 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.409101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.409242 (+ 141us) server_negotiation.cc:407] Connection header received
0928 15:01:49.409773 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.409774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.409785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.409883 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:49.411012 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.411605 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.412727 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.412912 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.412976 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.415055 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:49.415056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:49.415056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:49.415063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:49.416681 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.417061 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.417062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.417062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.417114 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.417322 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.417322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.417322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.417421 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:49.417470 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.417701 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.417762 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:01:49.417982 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.408966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35671 (local address 127.1.252.195:46903)
0928 15:01:49.409123 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.409165 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.409671 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.409950 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.409953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.410667 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:49.410902 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.410907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.411723 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.411724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.412620 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.412623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.413006 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.413007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.413126 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.413587 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:49.413595 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:49.414967 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:49.416754 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:49.416760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:49.416762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:49.416983 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:49.417171 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:49.417172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:49.417172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:49.417268 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:49.417473 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:49.417477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:49.417587 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.417668 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.417782 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":96,"threads_started":1}
I20260928 15:01:49.418509 20613 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35671
I20260928 15:01:49.418625 20613 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:49.418817 20613 heartbeater.cc:511] Master 127.1.252.254:35671 requested a full tablet report, sending...
I20260928 15:01:49.419194 20151 ts_manager.cc:194] Registered new tserver with Master: 1a7addfeaa174045b4e4e3487cfd745c (127.1.252.195:40957)
I20260928 15:01:49.419572 20151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:46903
I20260928 15:01:49.421893 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:49.429064 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.423256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.1.252.254:35671)
0928 15:01:49.423336 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.423338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.423404 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:01:49.423447 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.423448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.423456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.423502 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:49.424103 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.424575 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.425232 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.425408 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.425460 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.426655 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:49.426656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:49.426656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:49.426662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:49.428005 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.428369 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.428370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.428370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.428420 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:49.428602 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:49.428603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:49.428603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:49.428720 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:49.428799 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.428898 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.428946 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:01:49.430739 20151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:49.432559 20151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:49.438902 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.435064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:34147 (local address 127.0.0.1:33330)
0928 15:01:49.435548 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.435559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.435631 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.435977 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.435978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.435980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:49.436027 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.436028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.436891 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.436892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.437853 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.437862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.438699 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.438699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.438753 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.438764 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.438814 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.438850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":35,"thread_start_us":68,"threads_started":1}
I20260928 15:01:49.438957 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.435244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.1.252.194:34147)
0928 15:01:49.435419 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.435422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.435729 (+ 307us) server_negotiation.cc:407] Connection header received
0928 15:01:49.435780 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.435783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.435799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.435872 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:49.436117 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.436804 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.437960 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.438618 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.438694 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.438748 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.438838 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.438875 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260928 15:01:49.439311 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.434819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:37789 (local address 127.0.0.1:48532)
0928 15:01:49.434963 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.435006 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.435090 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.435381 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.435383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.435386 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:49.435453 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.435455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.436836 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.436836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.438204 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.438207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.439084 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.439084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.439141 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.439144 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.439194 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.439242 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":90,"thread_start_us":63,"threads_started":1}
I20260928 15:01:49.439539 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.434931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.1.252.193:37789)
0928 15:01:49.435129 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.435135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.435158 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:01:49.435198 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.435200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.435216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.435285 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:49.435566 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.436719 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.438295 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.439001 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.439084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.439142 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.439414 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.439461 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260928 15:01:49.439759 20410 tablet_service.cc:1467] Processing CreateTablet for tablet bb187c3c93644e64bf7439103c9080ba (DEFAULT_TABLE table=test-table [id=22a5df1bf3e8462a994320d5127a1ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:49.440142 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb187c3c93644e64bf7439103c9080ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:49.440248 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.434819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:40957 (local address 127.0.0.1:42822)
0928 15:01:49.435188 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.435199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.435252 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.435690 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.435691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.435692 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:49.435730 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.435732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.437026 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.437027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.438204 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.438207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.440001 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.440002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.440065 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.440070 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.440138 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.440181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":6,"spinlock_wait_cycles":384,"thread_start_us":81,"threads_started":1}
I20260928 15:01:49.440526 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.434992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.1.252.195:40957)
0928 15:01:49.435280 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.435285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.435295 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:01:49.435332 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.435334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.435349 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.435422 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:49.435806 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.436923 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.439182 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.439865 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.439997 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.440087 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.440155 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.440200 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":82,"threads_started":1}
I20260928 15:01:49.441100 20546 tablet_service.cc:1467] Processing CreateTablet for tablet bb187c3c93644e64bf7439103c9080ba (DEFAULT_TABLE table=test-table [id=22a5df1bf3e8462a994320d5127a1ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:49.441378 20546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb187c3c93644e64bf7439103c9080ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:49.442953 20630 tablet_bootstrap.cc:492] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Bootstrap starting.
I20260928 15:01:49.443430 20276 tablet_service.cc:1467] Processing CreateTablet for tablet bb187c3c93644e64bf7439103c9080ba (DEFAULT_TABLE table=test-table [id=22a5df1bf3e8462a994320d5127a1ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:49.443624 20630 tablet_bootstrap.cc:654] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:49.443749 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb187c3c93644e64bf7439103c9080ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:49.444031 20631 tablet_bootstrap.cc:492] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Bootstrap starting.
I20260928 15:01:49.444271 20630 log.cc:824] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:49.444813 20631 tablet_bootstrap.cc:654] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:49.445024 20630 tablet_bootstrap.cc:492] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: No bootstrap required, opened a new log
I20260928 15:01:49.445096 20630 ts_tablet_manager.cc:1397] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:49.445612 20631 log.cc:824] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:49.446374 20631 tablet_bootstrap.cc:492] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: No bootstrap required, opened a new log
I20260928 15:01:49.446445 20631 ts_tablet_manager.cc:1397] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:49.446957 20630 raft_consensus.cc:348] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.447124 20630 raft_consensus.cc:374] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:49.447165 20630 raft_consensus.cc:729] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125b0291de34d3d93458d04d9b1e6bf, State: Initialized, Role: FOLLOWER
I20260928 15:01:49.447296 20630 consensus_queue.cc:260] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.447604 20630 ts_tablet_manager.cc:1428] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:01:49.447731 20477 heartbeater.cc:503] Master 127.1.252.254:35671 was elected leader, sending a full tablet report...
I20260928 15:01:49.448104 20634 tablet_bootstrap.cc:492] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5: Bootstrap starting.
I20260928 15:01:49.448639 20634 tablet_bootstrap.cc:654] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:49.448669 20631 raft_consensus.cc:348] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.448830 20631 raft_consensus.cc:374] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:49.448866 20631 raft_consensus.cc:729] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7addfeaa174045b4e4e3487cfd745c, State: Initialized, Role: FOLLOWER
I20260928 15:01:49.448969 20631 consensus_queue.cc:260] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.449241 20631 ts_tablet_manager.cc:1428] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:49.449288 20634 log.cc:824] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:49.449329 20613 heartbeater.cc:503] Master 127.1.252.254:35671 was elected leader, sending a full tablet report...
I20260928 15:01:49.449976 20634 tablet_bootstrap.cc:492] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5: No bootstrap required, opened a new log
I20260928 15:01:49.450060 20634 ts_tablet_manager.cc:1397] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:49.451406 20634 raft_consensus.cc:348] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.451586 20634 raft_consensus.cc:374] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:49.451634 20634 raft_consensus.cc:729] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ca6004f9634ee7b76760ac970e81f5, State: Initialized, Role: FOLLOWER
I20260928 15:01:49.451742 20634 consensus_queue.cc:260] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.452054 20634 ts_tablet_manager.cc:1428] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:01:49.452075 20342 heartbeater.cc:503] Master 127.1.252.254:35671 was elected leader, sending a full tablet report...
W20260928 15:01:49.488490 20478 tablet.cc:2369] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:49.519789 20635 raft_consensus.cc:482] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:49.519941 20635 raft_consensus.cc:504] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.520300 20635 leader_election.cc:288] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00ca6004f9634ee7b76760ac970e81f5 (127.1.252.193:37789), 1a7addfeaa174045b4e4e3487cfd745c (127.1.252.195:40957)
I20260928 15:01:49.523393 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.520607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:52093 (local address 127.1.252.195:40957)
0928 15:01:49.520650 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.520652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.520779 (+ 127us) server_negotiation.cc:407] Connection header received
0928 15:01:49.520837 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.520839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.520848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.520939 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:49.521098 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.521569 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.522648 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.523041 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.523141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.523150 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.523287 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.523327 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:01:49.523454 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.520577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:59625 (local address 127.1.252.193:37789)
0928 15:01:49.520627 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.520630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.520642 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:01:49.520675 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.520676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.520686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.520765 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:49.520934 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.521416 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.522546 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.523035 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.523155 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.523164 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.523341 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.523381 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:01:49.523404 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.520540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:40957 (local address 127.1.252.194:52093)
0928 15:01:49.520706 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.520714 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.520782 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.520943 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.520944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.520945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:49.521022 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.521023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.521704 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.521705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.522559 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.522561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.523126 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.523127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.523199 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.523202 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.523268 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.523308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260928 15:01:49.523496 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.520491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:37789 (local address 127.1.252.194:59625)
0928 15:01:49.520550 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.520563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.520613 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.520789 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.520791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.520793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:49.520862 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.520864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.521549 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.521550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.522447 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.522449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.523170 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.523171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.523263 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.523266 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.523335 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.523377 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 15:01:49.523962 20296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb187c3c93644e64bf7439103c9080ba" candidate_uuid: "d125b0291de34d3d93458d04d9b1e6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00ca6004f9634ee7b76760ac970e81f5" is_pre_election: true
I20260928 15:01:49.523967 20567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb187c3c93644e64bf7439103c9080ba" candidate_uuid: "d125b0291de34d3d93458d04d9b1e6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7addfeaa174045b4e4e3487cfd745c" is_pre_election: true
I20260928 15:01:49.524116 20567 raft_consensus.cc:2454] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125b0291de34d3d93458d04d9b1e6bf in term 0.
I20260928 15:01:49.524132 20296 raft_consensus.cc:2454] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d125b0291de34d3d93458d04d9b1e6bf in term 0.
I20260928 15:01:49.524329 20365 leader_election.cc:302] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a7addfeaa174045b4e4e3487cfd745c, d125b0291de34d3d93458d04d9b1e6bf; no voters:
I20260928 15:01:49.524458 20635 raft_consensus.cc:2781] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:49.524521 20635 raft_consensus.cc:482] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:49.524544 20635 raft_consensus.cc:3037] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:49.525277 20635 raft_consensus.cc:504] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.525413 20635 leader_election.cc:288] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [CANDIDATE]: Term 1 election: Requested vote from peers 00ca6004f9634ee7b76760ac970e81f5 (127.1.252.193:37789), 1a7addfeaa174045b4e4e3487cfd745c (127.1.252.195:40957)
I20260928 15:01:49.525553 20296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb187c3c93644e64bf7439103c9080ba" candidate_uuid: "d125b0291de34d3d93458d04d9b1e6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00ca6004f9634ee7b76760ac970e81f5"
I20260928 15:01:49.525576 20567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb187c3c93644e64bf7439103c9080ba" candidate_uuid: "d125b0291de34d3d93458d04d9b1e6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7addfeaa174045b4e4e3487cfd745c"
I20260928 15:01:49.525635 20296 raft_consensus.cc:3037] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:49.525643 20567 raft_consensus.cc:3037] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:49.526489 20567 raft_consensus.cc:2454] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125b0291de34d3d93458d04d9b1e6bf in term 1.
I20260928 15:01:49.526489 20296 raft_consensus.cc:2454] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d125b0291de34d3d93458d04d9b1e6bf in term 1.
I20260928 15:01:49.526682 20366 leader_election.cc:302] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ca6004f9634ee7b76760ac970e81f5, d125b0291de34d3d93458d04d9b1e6bf; no voters:
I20260928 15:01:49.526777 20635 raft_consensus.cc:2781] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:49.526930 20635 raft_consensus.cc:686] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 1 LEADER]: Becoming Leader. State: Replica: d125b0291de34d3d93458d04d9b1e6bf, State: Running, Role: LEADER
I20260928 15:01:49.527006 20635 consensus_queue.cc:237] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } }
I20260928 15:01:49.527833 20151 catalog_manager.cc:5495] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf reported cstate change: term changed from 0 to 1, leader changed from <none> to d125b0291de34d3d93458d04d9b1e6bf (127.1.252.194). New cstate: current_term: 1 leader_uuid: "d125b0291de34d3d93458d04d9b1e6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125b0291de34d3d93458d04d9b1e6bf" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 34147 } health_report { overall_health: HEALTHY } } }
I20260928 15:01:49.556427 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.554057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.1.252.194:34147)
0928 15:01:49.554126 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.554128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.554142 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:01:49.554179 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.554179 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.554196 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.554258 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:49.554395 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.554886 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.555653 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.555843 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.555916 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.556051 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:49.556135 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:49.556205 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.556314 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.556329 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:01:49.558346 20296 raft_consensus.cc:1261] T bb187c3c93644e64bf7439103c9080ba P 00ca6004f9634ee7b76760ac970e81f5 [term 1 FOLLOWER]: Refusing update from remote peer d125b0291de34d3d93458d04d9b1e6bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:49.558346 20567 raft_consensus.cc:1261] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 1 FOLLOWER]: Refusing update from remote peer d125b0291de34d3d93458d04d9b1e6bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:49.558666 20635 consensus_queue.cc:1035] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ca6004f9634ee7b76760ac970e81f5" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:49.558809 20640 consensus_queue.cc:1035] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a7addfeaa174045b4e4e3487cfd745c" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:49.561236 20643 mvcc.cc:205] Tried to move back new op lower bound from 7334329178347933696 to 7334329178223394816. Current Snapshot: MvccSnapshot[applied={T|T < 7334329178347933696}]
I20260928 15:01:49.561497 20644 mvcc.cc:205] Tried to move back new op lower bound from 7334329178347933696 to 7334329178223394816. Current Snapshot: MvccSnapshot[applied={T|T < 7334329178347933696}]
I20260928 15:01:49.562450 20645 mvcc.cc:205] Tried to move back new op lower bound from 7334329178347933696 to 7334329178223394816. Current Snapshot: MvccSnapshot[applied={T|T < 7334329178347933696 or (T in {7334329178347933696})}]
I20260928 15:01:49.564459 20151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:49.564570 20151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:49.565310 20151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86e84404073b4bcfa73dac8a2be6d5bb: Sending DeleteTablet for 3 replicas of tablet bb187c3c93644e64bf7439103c9080ba
I20260928 15:01:49.565625 20276 tablet_service.cc:1514] Processing DeleteTablet for tablet bb187c3c93644e64bf7439103c9080ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:49 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:48532
I20260928 15:01:49.565649 20546 tablet_service.cc:1514] Processing DeleteTablet for tablet bb187c3c93644e64bf7439103c9080ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:49 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:42822
I20260928 15:01:49.565968 20410 tablet_service.cc:1514] Processing DeleteTablet for tablet bb187c3c93644e64bf7439103c9080ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:49 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:33330
I20260928 15:01:49.566061 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20213
I20260928 15:01:49.566229 20655 tablet_replica.cc:330] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: stopping tablet replica
I20260928 15:01:49.566321 20655 raft_consensus.cc:2229] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:49.566433 20655 raft_consensus.cc:2258] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:49.566901 20655 ts_tablet_manager.cc:1905] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:49.568771 20655 ts_tablet_manager.cc:1918] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:49.568837 20655 log.cc:1196] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/bb187c3c93644e64bf7439103c9080ba
I20260928 15:01:49.569108 20655 ts_tablet_manager.cc:1939] T bb187c3c93644e64bf7439103c9080ba P d125b0291de34d3d93458d04d9b1e6bf: Deleting consensus metadata
I20260928 15:01:49.569290 20654 tablet_replica.cc:330] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: stopping tablet replica
I20260928 15:01:49.569391 20654 raft_consensus.cc:2229] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:49.569540 20654 raft_consensus.cc:2258] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:49.569996 20654 ts_tablet_manager.cc:1905] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:49.569990 20135 catalog_manager.cc:4841] TS d125b0291de34d3d93458d04d9b1e6bf (127.1.252.194:34147): tablet bb187c3c93644e64bf7439103c9080ba (table test-table [id=22a5df1bf3e8462a994320d5127a1ee8]) successfully deleted
I20260928 15:01:49.571617 20654 ts_tablet_manager.cc:1918] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:49.571692 20654 log.cc:1196] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/bb187c3c93644e64bf7439103c9080ba
I20260928 15:01:49.572003 20654 ts_tablet_manager.cc:1939] T bb187c3c93644e64bf7439103c9080ba P 1a7addfeaa174045b4e4e3487cfd745c: Deleting consensus metadata
W20260928 15:01:49.572405 20138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:37789 (error 108)
I20260928 15:01:49.572661 20137 catalog_manager.cc:4841] TS 1a7addfeaa174045b4e4e3487cfd745c (127.1.252.195:40957): tablet bb187c3c93644e64bf7439103c9080ba (table test-table [id=22a5df1bf3e8462a994320d5127a1ee8]) successfully deleted
I20260928 15:01:49.572691 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20347
I20260928 15:01:49.573307 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.572812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:37789 (local address 127.0.0.1:48544)
0928 15:01:49.573228 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.573268 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:37789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260928 15:01:49.573486 20138 catalog_manager.cc:4584] TS 00ca6004f9634ee7b76760ac970e81f5 (127.1.252.193:37789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb187c3c93644e64bf7439103c9080ba: Network error: Client connection negotiation failed: client connection to 127.1.252.193:37789: connect: Connection refused (error 111)
I20260928 15:01:49.577620 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20482
I20260928 15:01:49.582125 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20119
2026-09-28T15:01:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T15:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:49Z Disabled control of system clock
I20260928 15:01:49.610112 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33395
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:49.690517 20660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:49.690727 20660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:49.690750 20660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:49.691810 20660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:49.691847 20660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:49.691859 20660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:49.691872 20660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:49.691931 20660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:49.693243 20660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:33395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:49.693449 20660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:49.693683 20660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:49.699038 20668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.699069 20666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.699069 20665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.699329 20660 server_base.cc:1034] running on GCE node
I20260928 15:01:49.699640 20660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:49.699970 20660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:49.701144 20660 hybrid_clock.cc:648] HybridClock initialized: now 1790607709701115 us; error 45 us; skew 500 ppm
I20260928 15:01:49.702534 20660 webserver.cc:460] Webserver started at http://127.1.252.254:32781/ using document root <none> and password file <none>
I20260928 15:01:49.702811 20660 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:49.702859 20660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:49.703023 20660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:49.704171 20660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "33562e125ef04535b96f194083341632"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "1ef2de9cb3c61f4168dd3fcd1ec5be36"
server_key_iv: "3857f2de77c391c21cb8bbdb914991c4"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.704540 20660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "33562e125ef04535b96f194083341632"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "1ef2de9cb3c61f4168dd3fcd1ec5be36"
server_key_iv: "3857f2de77c391c21cb8bbdb914991c4"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.705866 20660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:01:49.706598 20674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.706779 20660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:49.706849 20660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "33562e125ef04535b96f194083341632"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "1ef2de9cb3c61f4168dd3fcd1ec5be36"
server_key_iv: "3857f2de77c391c21cb8bbdb914991c4"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.706944 20660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:49.719848 20660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:49.720242 20660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:49.720395 20660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:49.724347 20660 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:33395
I20260928 15:01:49.724407 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:33395 every 8 connection(s)
I20260928 15:01:49.724722 20660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:49.725680 20727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:49.728005 20727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632: Bootstrap starting.
I20260928 15:01:49.728586 20727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:49.728881 20727 log.cc:824] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:49.729615 20727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632: No bootstrap required, opened a new log
I20260928 15:01:49.731182 20727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33562e125ef04535b96f194083341632" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33395 } }
I20260928 15:01:49.731341 20727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:49.731387 20727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33562e125ef04535b96f194083341632, State: Initialized, Role: FOLLOWER
I20260928 15:01:49.731564 20727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33562e125ef04535b96f194083341632" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33395 } }
I20260928 15:01:49.731650 20727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:49.731693 20727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:49.731742 20727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:49.732333 20727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33562e125ef04535b96f194083341632" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33395 } }
I20260928 15:01:49.732447 20727 leader_election.cc:302] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33562e125ef04535b96f194083341632; no voters:
I20260928 15:01:49.732630 20727 leader_election.cc:288] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:49.732683 20730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:49.732887 20727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:49.732889 20730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [term 1 LEADER]: Becoming Leader. State: Replica: 33562e125ef04535b96f194083341632, State: Running, Role: LEADER
I20260928 15:01:49.733068 20730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33562e125ef04535b96f194083341632" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33395 } }
I20260928 15:01:49.733537 20731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33562e125ef04535b96f194083341632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33562e125ef04535b96f194083341632" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33395 } } }
I20260928 15:01:49.733619 20732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33562e125ef04535b96f194083341632. Latest consensus state: current_term: 1 leader_uuid: "33562e125ef04535b96f194083341632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33562e125ef04535b96f194083341632" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33395 } } }
I20260928 15:01:49.733640 20731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:49.733667 20732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:49.733850 20739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:49.734337 20739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:49.734402 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20660
I20260928 15:01:49.734515 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:49.734751 2035 external_mini_cluster.cc:1428] Setting key 34d8f4b699ec356b42f715e734ef941c
I20260928 15:01:49.736400 20739 catalog_manager.cc:1347] Generated new cluster ID: 058564a9a8294108995068a5caddb1c1
I20260928 15:01:49.736459 20739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:49.739807 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.735344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.1.252.254:33395)
0928 15:01:49.735556 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.735564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.735596 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:01:49.735707 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.735713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.735779 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.735965 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:49.736158 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.737165 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.737924 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.738352 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.738502 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.739176 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:49.739180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:49.739180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:49.739355 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:49.739386 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:49.739397 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:49.739474 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.739548 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.739620 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260928 15:01:49.743911 20739 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:49.744377 20739 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:49.747859 20739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632: Generated new TSK 0
I20260928 15:01:49.748056 20739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:49.749449 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33395
--builtin_ntp_servers=127.1.252.212:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:49.843499 20751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:49.843704 20751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:49.843727 20751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:49.845203 20751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:49.845253 20751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:49.845281 20751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:49.846743 20751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:49.847004 20751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:49.847323 20751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:49.853258 20756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.853271 20757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.853498 20751 server_base.cc:1034] running on GCE node
W20260928 15:01:49.853271 20759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.853843 20751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:49.854120 20751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:49.855305 20751 hybrid_clock.cc:648] HybridClock initialized: now 1790607709855274 us; error 48 us; skew 500 ppm
I20260928 15:01:49.863827 20751 webserver.cc:460] Webserver started at http://127.1.252.193:45037/ using document root <none> and password file <none>
I20260928 15:01:49.864130 20751 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:49.864197 20751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:49.864322 20751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:49.865842 20751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "a802aef1ae434cb788435a50346e2702"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "f7b5b192db2fb76628e289c5e6fbe3aa"
server_key_iv: "c3a8686d6d044813d949992107391cdc"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.866237 20751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "a802aef1ae434cb788435a50346e2702"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "f7b5b192db2fb76628e289c5e6fbe3aa"
server_key_iv: "c3a8686d6d044813d949992107391cdc"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.867861 20751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 15:01:49.868889 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.869155 20751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:49.869251 20751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "a802aef1ae434cb788435a50346e2702"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "f7b5b192db2fb76628e289c5e6fbe3aa"
server_key_iv: "c3a8686d6d044813d949992107391cdc"
server_key_version: "encryptionkey@0"
I20260928 15:01:49.869323 20751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:49.892722 20751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:49.893008 20751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:49.893112 20751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:49.893308 20751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:49.893668 20751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:49.893704 20751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.893760 20751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:49.893787 20751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:49.899693 20751 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:36715
I20260928 15:01:49.899763 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:36715 every 8 connection(s)
I20260928 15:01:49.900113 20751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:49.905413 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20751
I20260928 15:01:49.905542 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:49.905805 2035 external_mini_cluster.cc:1428] Setting key dd9f9bb8f1059d4c02c8a3efccd1c980
I20260928 15:01:49.905977 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.900932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:53173 (local address 127.1.252.254:33395)
0928 15:01:49.900980 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:49.900982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:49.901196 (+ 214us) server_negotiation.cc:407] Connection header received
0928 15:01:49.901676 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:49.901677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:49.901690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:49.901759 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:49.902435 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.902973 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.904100 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.904301 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.904358 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.905663 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:49.905664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:49.905665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:49.905672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:49.905690 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:49.905696 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:49.905765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:49.905864 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 15:01:49.905898 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:01:49.906224 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:49.900884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33395 (local address 127.1.252.193:53173)
0928 15:01:49.901065 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:49.901105 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:49.901582 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:49.901827 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:49.901830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:49.901838 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:49.902354 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.902358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.903058 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.903059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.904018 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:49.904021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:49.904385 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:49.904386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:49.904504 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:49.905387 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:49.905387 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:49.905466 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:49.905489 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:49.905771 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:49.905776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:49.905778 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:49.905835 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 15:01:49.905960 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 15:01:49.906875 20880 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33395
I20260928 15:01:49.906955 20880 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:49.906987 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33395
--builtin_ntp_servers=127.1.252.212:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:49.907167 20880 heartbeater.cc:511] Master 127.1.252.254:33395 requested a full tablet report, sending...
I20260928 15:01:49.907717 20691 ts_manager.cc:194] Registered new tserver with Master: a802aef1ae434cb788435a50346e2702 (127.1.252.193:36715)
I20260928 15:01:49.908638 20691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.193:53173
W20260928 15:01:49.988303 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:49.988528 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:49.988564 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:49.989682 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:49.989741 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:49.989784 20883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:49.991117 20883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:49.991370 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:49.991660 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:49.996696 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.996711 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:49.996695 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:49.996959 20883 server_base.cc:1034] running on GCE node
I20260928 15:01:49.997181 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:49.997430 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:49.998618 20883 hybrid_clock.cc:648] HybridClock initialized: now 1790607709998610 us; error 39 us; skew 500 ppm
I20260928 15:01:50.000103 20883 webserver.cc:460] Webserver started at http://127.1.252.194:44613/ using document root <none> and password file <none>
I20260928 15:01:50.000370 20883 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:50.000449 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:50.000603 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:50.001686 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "e29b4bd9de89400c9a76c80e9392b4ec"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "059a5efcac793cdb8aac1c38e7b4d93a"
server_key_iv: "134f6705896b04497a4933a9769b3692"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.002058 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "e29b4bd9de89400c9a76c80e9392b4ec"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "059a5efcac793cdb8aac1c38e7b4d93a"
server_key_iv: "134f6705896b04497a4933a9769b3692"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.003508 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:50.004431 20897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.004706 20883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:50.004844 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "e29b4bd9de89400c9a76c80e9392b4ec"
format_stamp: "Formatted at 2026-09-28 15:01:49 on dist-test-slave-sf70"
server_key: "059a5efcac793cdb8aac1c38e7b4d93a"
server_key_iv: "134f6705896b04497a4933a9769b3692"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.004930 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:50.012538 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:50.012859 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:50.013025 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:50.013257 20883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:50.013621 20883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:50.013706 20883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.013761 20883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:50.013793 20883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.020434 20883 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:40825
I20260928 15:01:50.020471 21010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:40825 every 8 connection(s)
I20260928 15:01:50.020815 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:50.022218 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 20883
I20260928 15:01:50.022300 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:50.022527 2035 external_mini_cluster.cc:1428] Setting key 2fb074d6865316f1a0863612cd9ef310
I20260928 15:01:50.023717 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33395
--builtin_ntp_servers=127.1.252.212:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:50.026969 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.021637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:47325 (local address 127.1.252.254:33395)
0928 15:01:50.021705 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.021707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.021947 (+ 240us) server_negotiation.cc:407] Connection header received
0928 15:01:50.022552 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.022554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.022567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.022647 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:50.023591 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.024248 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.025564 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.025765 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.025840 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.026681 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:50.026683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:50.026683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:50.026687 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:50.026704 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:50.026709 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:50.026765 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.026828 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.026855 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:01:50.027114 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.021614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33395 (local address 127.1.252.194:47325)
0928 15:01:50.021813 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.021855 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.022440 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.022790 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.022793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.022801 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:50.023461 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.023468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.024339 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.024340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.025461 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.025464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.025847 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.025848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.025995 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.026543 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:50.026543 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:50.026570 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:50.026592 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:50.026772 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:50.026776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:50.026779 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.026829 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.026912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260928 15:01:50.027618 21011 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33395
I20260928 15:01:50.027709 21011 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:50.027840 21011 heartbeater.cc:511] Master 127.1.252.254:33395 requested a full tablet report, sending...
I20260928 15:01:50.028228 20691 ts_manager.cc:194] Registered new tserver with Master: e29b4bd9de89400c9a76c80e9392b4ec (127.1.252.194:40825)
I20260928 15:01:50.028764 20691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.194:47325
W20260928 15:01:50.105845 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:50.106045 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:50.106066 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:50.107165 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:50.107209 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:50.107254 21014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:50.108682 21014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:33395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:50.108954 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:50.109203 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:50.114274 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:50.114319 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:50.114284 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:50.114425 21014 server_base.cc:1034] running on GCE node
I20260928 15:01:50.114827 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:50.115083 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:50.116241 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790607710116224 us; error 26 us; skew 500 ppm
I20260928 15:01:50.117587 21014 webserver.cc:460] Webserver started at http://127.1.252.195:38425/ using document root <none> and password file <none>
I20260928 15:01:50.117839 21014 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:50.117885 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:50.118073 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:50.119189 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "ef73bc0f6f994b108d63458aa1e6078f"
format_stamp: "Formatted at 2026-09-28 15:01:50 on dist-test-slave-sf70"
server_key: "157c473e48e28c0106b6df629a7d2bd7"
server_key_iv: "f73661201345a639ac24e9424351c5b3"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.119551 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "ef73bc0f6f994b108d63458aa1e6078f"
format_stamp: "Formatted at 2026-09-28 15:01:50 on dist-test-slave-sf70"
server_key: "157c473e48e28c0106b6df629a7d2bd7"
server_key_iv: "f73661201345a639ac24e9424351c5b3"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.120906 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:50.121721 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.121961 21014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:50.122038 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "ef73bc0f6f994b108d63458aa1e6078f"
format_stamp: "Formatted at 2026-09-28 15:01:50 on dist-test-slave-sf70"
server_key: "157c473e48e28c0106b6df629a7d2bd7"
server_key_iv: "f73661201345a639ac24e9424351c5b3"
server_key_version: "encryptionkey@0"
I20260928 15:01:50.122143 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:50.133992 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:50.134289 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:50.134394 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:50.134649 21014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:50.135005 21014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:50.135061 21014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.135103 21014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:50.135131 21014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:50.141024 21014 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:41457
I20260928 15:01:50.141108 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:41457 every 8 connection(s)
I20260928 15:01:50.141562 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:50.146883 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.142366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:59621 (local address 127.1.252.254:33395)
0928 15:01:50.142448 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.142450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.142648 (+ 198us) server_negotiation.cc:407] Connection header received
0928 15:01:50.143143 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.143144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.143157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.143233 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:50.143958 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.144493 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.145497 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.145684 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.145740 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.146469 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:50.146470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:50.146470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:50.146476 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:50.146490 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:50.146494 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:50.146574 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.146771 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.146795 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:01:50.147033 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.142321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:33395 (local address 127.1.252.195:59621)
0928 15:01:50.142501 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.142541 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.143018 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.143299 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.143303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.143310 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:50.143801 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.143804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.144601 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.144602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.145412 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.145414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.145760 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.145761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.145871 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.146345 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:50.146345 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:50.146372 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:50.146394 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:50.146666 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:50.146675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:50.146679 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.146762 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.146842 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 15:01:50.147809 21142 heartbeater.cc:347] Connected to a master server at 127.1.252.254:33395
I20260928 15:01:50.147990 21142 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:50.148193 21142 heartbeater.cc:511] Master 127.1.252.254:33395 requested a full tablet report, sending...
I20260928 15:01:50.148627 20691 ts_manager.cc:194] Registered new tserver with Master: ef73bc0f6f994b108d63458aa1e6078f (127.1.252.195:41457)
I20260928 15:01:50.149058 20691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.195:59621
I20260928 15:01:50.149421 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21014
I20260928 15:01:50.149525 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:50.149740 2035 external_mini_cluster.cc:1428] Setting key 3f566d1462c8a62b2c9cf548b05701fd
I20260928 15:01:50.150738 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:50.154929 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.152144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.1.252.254:33395)
0928 15:01:50.152187 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.152189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.152245 (+ 56us) server_negotiation.cc:407] Connection header received
0928 15:01:50.152295 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.152295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.152304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.152355 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:50.152471 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.152958 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.153764 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.153938 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.153994 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.154699 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:50.154700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:50.154700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:50.154704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:50.154717 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:50.154720 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:50.154796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.154868 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.154877 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:01:50.156798 20691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:50.158643 20691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:50.165402 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.160942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:41457 (local address 127.0.0.1:59868)
0928 15:01:50.161437 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.161447 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.161500 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.161711 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.161714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.161716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:50.161793 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.161794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.163253 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.163253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.164389 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.164392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.165100 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.165101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.165178 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.165191 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.165259 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.165313 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":23,"thread_start_us":83,"threads_started":1}
I20260928 15:01:50.165493 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.161215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.1.252.194:40825)
0928 15:01:50.161388 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.161392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.162033 (+ 641us) server_negotiation.cc:407] Connection header received
0928 15:01:50.162094 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.162096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.162111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.162205 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:50.162297 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.162967 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.164492 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.165191 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.165249 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.165322 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.165356 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.165393 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260928 15:01:50.165660 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.161071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.1.252.193:36715)
0928 15:01:50.161591 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.161595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.161603 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:01:50.161646 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.161647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.161680 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.161752 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:50.161915 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.163092 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.164347 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.165119 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.165192 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.165283 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.165530 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.165578 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":77,"threads_started":1}
I20260928 15:01:50.165750 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.161189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.1.252.195:41457)
0928 15:01:50.161354 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.161358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.161491 (+ 133us) server_negotiation.cc:407] Connection header received
0928 15:01:50.161587 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.161589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.161605 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.161692 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:50.161938 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.163092 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.164493 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.165007 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.165100 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.165198 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.165633 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.165676 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260928 15:01:50.166015 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.160981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:40825 (local address 127.0.0.1:57790)
0928 15:01:50.161952 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.161960 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.162022 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.162201 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.162201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.162203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:50.162238 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.162239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.163065 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.163066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.164412 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.164414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.165251 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.165252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.165307 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.165309 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.165348 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.165381 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260928 15:01:50.166486 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 82559ffbbd314bea811ef0f8abfc2df2 (DEFAULT_TABLE table=test-table [id=ef941ea07b464e7780d8b060409666ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:50.166837 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82559ffbbd314bea811ef0f8abfc2df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:50.168032 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.160942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36715 (local address 127.0.0.1:44186)
0928 15:01:50.161269 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.161313 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.161398 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.161816 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.161816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.161818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:50.161854 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.161856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.163163 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.163164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.164259 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.164262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.165321 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.165322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.165388 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.165392 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.165442 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.165485 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":44,"thread_start_us":68,"threads_started":1}
I20260928 15:01:50.168869 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 82559ffbbd314bea811ef0f8abfc2df2 (DEFAULT_TABLE table=test-table [id=ef941ea07b464e7780d8b060409666ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:50.169025 20814 tablet_service.cc:1467] Processing CreateTablet for tablet 82559ffbbd314bea811ef0f8abfc2df2 (DEFAULT_TABLE table=test-table [id=ef941ea07b464e7780d8b060409666ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:50.169273 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82559ffbbd314bea811ef0f8abfc2df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:50.169358 20814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82559ffbbd314bea811ef0f8abfc2df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:50.170037 21156 tablet_bootstrap.cc:492] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Bootstrap starting.
I20260928 15:01:50.170801 21156 tablet_bootstrap.cc:654] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:50.171160 21156 log.cc:824] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:50.171990 21156 tablet_bootstrap.cc:492] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: No bootstrap required, opened a new log
I20260928 15:01:50.172066 21156 ts_tablet_manager.cc:1397] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:50.172529 21158 tablet_bootstrap.cc:492] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: Bootstrap starting.
I20260928 15:01:50.172803 21159 tablet_bootstrap.cc:492] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Bootstrap starting.
I20260928 15:01:50.173221 21158 tablet_bootstrap.cc:654] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:50.173442 21159 tablet_bootstrap.cc:654] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:50.173548 21158 log.cc:824] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:50.173810 21159 log.cc:824] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:50.174326 21158 tablet_bootstrap.cc:492] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: No bootstrap required, opened a new log
I20260928 15:01:50.174441 21158 ts_tablet_manager.cc:1397] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:50.174571 21159 tablet_bootstrap.cc:492] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: No bootstrap required, opened a new log
I20260928 15:01:50.174639 21159 ts_tablet_manager.cc:1397] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:50.175040 21156 raft_consensus.cc:348] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.175173 21156 raft_consensus.cc:374] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:50.175194 21156 raft_consensus.cc:729] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef73bc0f6f994b108d63458aa1e6078f, State: Initialized, Role: FOLLOWER
I20260928 15:01:50.175309 21156 consensus_queue.cc:260] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.175581 21156 ts_tablet_manager.cc:1428] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:50.175732 21142 heartbeater.cc:503] Master 127.1.252.254:33395 was elected leader, sending a full tablet report...
I20260928 15:01:50.177238 21159 raft_consensus.cc:348] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.177301 21158 raft_consensus.cc:348] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.177404 21159 raft_consensus.cc:374] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:50.177423 21158 raft_consensus.cc:374] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:50.177440 21159 raft_consensus.cc:729] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29b4bd9de89400c9a76c80e9392b4ec, State: Initialized, Role: FOLLOWER
I20260928 15:01:50.177455 21158 raft_consensus.cc:729] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a802aef1ae434cb788435a50346e2702, State: Initialized, Role: FOLLOWER
I20260928 15:01:50.177563 21159 consensus_queue.cc:260] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.177563 21158 consensus_queue.cc:260] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.177819 21159 ts_tablet_manager.cc:1428] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:50.177819 21158 ts_tablet_manager.cc:1428] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:50.177886 20880 heartbeater.cc:503] Master 127.1.252.254:33395 was elected leader, sending a full tablet report...
I20260928 15:01:50.177948 21011 heartbeater.cc:503] Master 127.1.252.254:33395 was elected leader, sending a full tablet report...
W20260928 15:01:50.271482 21012 tablet.cc:2369] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:50.392227 21143 tablet.cc:2369] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:50.400853 20881 tablet.cc:2369] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:50.551492 21162 raft_consensus.cc:482] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:50.551689 21162 raft_consensus.cc:504] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.552160 21162 leader_election.cc:288] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a802aef1ae434cb788435a50346e2702 (127.1.252.193:36715), e29b4bd9de89400c9a76c80e9392b4ec (127.1.252.194:40825)
I20260928 15:01:50.555076 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.552428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:43545 (local address 127.1.252.193:36715)
0928 15:01:50.552479 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.552482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.552669 (+ 187us) server_negotiation.cc:407] Connection header received
0928 15:01:50.552731 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.552732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.552745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.552814 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:50.552968 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.553429 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.554369 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.554720 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.554805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.554820 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.554969 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.555001 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:01:50.555110 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.552354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:40825 (local address 127.1.252.195:44023)
0928 15:01:50.552441 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.552453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.552498 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.552712 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.552713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.552715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:50.552793 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.552795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.553461 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.553462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.554275 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.554277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.554835 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.554836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.554905 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.554908 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.554974 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.555025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":18}
I20260928 15:01:50.555095 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.552506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:44023 (local address 127.1.252.194:40825)
0928 15:01:50.552562 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.552564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.552573 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:01:50.552606 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.552607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.552625 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.552706 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:50.552867 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.553359 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.554370 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.554727 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.554819 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.554828 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.554981 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.555020 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 15:01:50.555133 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.552366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36715 (local address 127.1.252.195:43545)
0928 15:01:50.552588 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.552597 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:50.552675 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:50.552817 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:50.552818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:50.552820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:50.552890 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.552891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.553515 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.553516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.554275 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:50.554277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.554805 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.554806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:50.554875 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.554879 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:50.554960 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:01:50.555000 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":22,"thread_start_us":112,"threads_started":1}
I20260928 15:01:50.555442 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82559ffbbd314bea811ef0f8abfc2df2" candidate_uuid: "ef73bc0f6f994b108d63458aa1e6078f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" is_pre_election: true
I20260928 15:01:50.555478 20834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82559ffbbd314bea811ef0f8abfc2df2" candidate_uuid: "ef73bc0f6f994b108d63458aa1e6078f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a802aef1ae434cb788435a50346e2702" is_pre_election: true
I20260928 15:01:50.555603 20965 raft_consensus.cc:2454] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef73bc0f6f994b108d63458aa1e6078f in term 0.
I20260928 15:01:50.555621 20834 raft_consensus.cc:2454] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef73bc0f6f994b108d63458aa1e6078f in term 0.
I20260928 15:01:50.555851 21030 leader_election.cc:302] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a802aef1ae434cb788435a50346e2702, ef73bc0f6f994b108d63458aa1e6078f; no voters:
I20260928 15:01:50.556015 21162 raft_consensus.cc:2781] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:50.556104 21162 raft_consensus.cc:482] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:50.556142 21162 raft_consensus.cc:3037] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:50.556857 21162 raft_consensus.cc:504] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.557005 21162 leader_election.cc:288] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [CANDIDATE]: Term 1 election: Requested vote from peers a802aef1ae434cb788435a50346e2702 (127.1.252.193:36715), e29b4bd9de89400c9a76c80e9392b4ec (127.1.252.194:40825)
I20260928 15:01:50.557204 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82559ffbbd314bea811ef0f8abfc2df2" candidate_uuid: "ef73bc0f6f994b108d63458aa1e6078f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29b4bd9de89400c9a76c80e9392b4ec"
I20260928 15:01:50.557232 20834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82559ffbbd314bea811ef0f8abfc2df2" candidate_uuid: "ef73bc0f6f994b108d63458aa1e6078f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a802aef1ae434cb788435a50346e2702"
I20260928 15:01:50.557288 20965 raft_consensus.cc:3037] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:50.557294 20834 raft_consensus.cc:3037] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:50.558185 20834 raft_consensus.cc:2454] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef73bc0f6f994b108d63458aa1e6078f in term 1.
I20260928 15:01:50.558197 20965 raft_consensus.cc:2454] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef73bc0f6f994b108d63458aa1e6078f in term 1.
I20260928 15:01:50.558368 21031 leader_election.cc:302] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e29b4bd9de89400c9a76c80e9392b4ec, ef73bc0f6f994b108d63458aa1e6078f; no voters:
I20260928 15:01:50.558478 21162 raft_consensus.cc:2781] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:50.558632 21162 raft_consensus.cc:686] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 1 LEADER]: Becoming Leader. State: Replica: ef73bc0f6f994b108d63458aa1e6078f, State: Running, Role: LEADER
I20260928 15:01:50.558734 21162 consensus_queue.cc:237] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } }
I20260928 15:01:50.559615 20690 catalog_manager.cc:5495] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f reported cstate change: term changed from 0 to 1, leader changed from <none> to ef73bc0f6f994b108d63458aa1e6078f (127.1.252.195). New cstate: current_term: 1 leader_uuid: "ef73bc0f6f994b108d63458aa1e6078f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef73bc0f6f994b108d63458aa1e6078f" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 41457 } health_report { overall_health: HEALTHY } } }
I20260928 15:01:50.605681 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.603280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.1.252.195:41457)
0928 15:01:50.603343 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:50.603345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:50.603353 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:01:50.603462 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:50.603463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:50.603483 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:50.603543 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:50.603669 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.604234 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.604958 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:50.605143 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:50.605198 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:50.605326 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:50.605416 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:50.605486 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:50.605537 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 15:01:50.605546 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:01:50.607481 20965 raft_consensus.cc:1261] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 1 FOLLOWER]: Refusing update from remote peer ef73bc0f6f994b108d63458aa1e6078f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:50.607486 20834 raft_consensus.cc:1261] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 1 FOLLOWER]: Refusing update from remote peer ef73bc0f6f994b108d63458aa1e6078f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:50.607838 21162 consensus_queue.cc:1035] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e29b4bd9de89400c9a76c80e9392b4ec" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:50.608026 21167 consensus_queue.cc:1035] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a802aef1ae434cb788435a50346e2702" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 36715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:50.610251 21172 mvcc.cc:205] Tried to move back new op lower bound from 7334329182645334016 to 7334329182449127424. Current Snapshot: MvccSnapshot[applied={T|T < 7334329182645334016}]
I20260928 15:01:50.611351 21171 mvcc.cc:205] Tried to move back new op lower bound from 7334329182645334016 to 7334329182449127424. Current Snapshot: MvccSnapshot[applied={T|T < 7334329182645334016}]
I20260928 15:01:50.612834 20691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:50.612921 20691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:50.613685 20691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33562e125ef04535b96f194083341632: Sending DeleteTablet for 3 replicas of tablet 82559ffbbd314bea811ef0f8abfc2df2
I20260928 15:01:50.613991 20814 tablet_service.cc:1514] Processing DeleteTablet for tablet 82559ffbbd314bea811ef0f8abfc2df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:50 UTC) from {username='slave'} at 127.0.0.1:44186
I20260928 15:01:50.614035 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 82559ffbbd314bea811ef0f8abfc2df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:50 UTC) from {username='slave'} at 127.0.0.1:57790
I20260928 15:01:50.614251 21180 tablet_replica.cc:330] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: stopping tablet replica
I20260928 15:01:50.614276 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 82559ffbbd314bea811ef0f8abfc2df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:50 UTC) from {username='slave'} at 127.0.0.1:59868
I20260928 15:01:50.614358 21180 raft_consensus.cc:2229] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:50.614476 21179 tablet_replica.cc:330] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702: stopping tablet replica
I20260928 15:01:50.614555 21180 raft_consensus.cc:2258] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:50.614557 21179 raft_consensus.cc:2229] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:50.614569 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20751
I20260928 15:01:50.614655 21179 raft_consensus.cc:2258] T 82559ffbbd314bea811ef0f8abfc2df2 P a802aef1ae434cb788435a50346e2702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:50.614787 21181 tablet_replica.cc:330] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: stopping tablet replica
I20260928 15:01:50.614869 21181 raft_consensus.cc:2229] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:50.614964 21181 raft_consensus.cc:2258] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:50.614997 21180 ts_tablet_manager.cc:1905] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:50.615649 21181 ts_tablet_manager.cc:1905] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:50.616843 21180 ts_tablet_manager.cc:1918] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:50.616904 21180 log.cc:1196] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/82559ffbbd314bea811ef0f8abfc2df2
I20260928 15:01:50.617147 21180 ts_tablet_manager.cc:1939] T 82559ffbbd314bea811ef0f8abfc2df2 P e29b4bd9de89400c9a76c80e9392b4ec: Deleting consensus metadata
I20260928 15:01:50.617353 21181 ts_tablet_manager.cc:1918] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:50.617408 21181 log.cc:1196] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/82559ffbbd314bea811ef0f8abfc2df2
I20260928 15:01:50.617552 20677 catalog_manager.cc:4841] TS e29b4bd9de89400c9a76c80e9392b4ec (127.1.252.194:40825): tablet 82559ffbbd314bea811ef0f8abfc2df2 (table test-table [id=ef941ea07b464e7780d8b060409666ff]) successfully deleted
I20260928 15:01:50.617799 21181 ts_tablet_manager.cc:1939] T 82559ffbbd314bea811ef0f8abfc2df2 P ef73bc0f6f994b108d63458aa1e6078f: Deleting consensus metadata
I20260928 15:01:50.618126 20675 catalog_manager.cc:4841] TS ef73bc0f6f994b108d63458aa1e6078f (127.1.252.195:41457): tablet 82559ffbbd314bea811ef0f8abfc2df2 (table test-table [id=ef941ea07b464e7780d8b060409666ff]) successfully deleted
W20260928 15:01:50.623015 20676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:36715 (error 108)
I20260928 15:01:50.623281 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20883
I20260928 15:01:50.623504 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:50.623390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:36715 (local address 127.0.0.1:44194)
0928 15:01:50.623427 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:50.623466 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:36715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 15:01:50.624078 20676 catalog_manager.cc:4584] TS a802aef1ae434cb788435a50346e2702 (127.1.252.193:36715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82559ffbbd314bea811ef0f8abfc2df2: Network error: Client connection negotiation failed: client connection to 127.1.252.193:36715: connect: Connection refused (error 111)
I20260928 15:01:50.627604 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21014
I20260928 15:01:50.631623 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 20660
2026-09-28T15:01:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1047 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:50 dist-test-slave-sf70 krb5kdc[21185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:50 dist-test-slave-sf70 krb5kdc[21185](info): set up 2 sockets
Sep 28 15:01:50 dist-test-slave-sf70 krb5kdc[21185](info): commencing operation
krb5kdc: starting...
W20260928 15:01:51.673426 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:51 dist-test-slave-sf70 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:01:51.794767 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42477
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:42477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:51.874864 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:51.875070 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:51.875092 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:51.876183 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:51.876219 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:51.876232 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:51.876245 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:51.876256 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:51.877637 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:42477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:42477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:51.877826 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:51.877985 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:51.883213 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:51.883268 21198 server_base.cc:1034] running on GCE node
W20260928 15:01:51.883232 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:51.883229 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:51.883730 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:51.884037 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:51.885174 21198 hybrid_clock.cc:648] HybridClock initialized: now 1790607711885159 us; error 28 us; skew 500 ppm
Sep 28 15:01:51 dist-test-slave-sf70 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607711, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:51.887715 21198 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:01:51.888190 21198 webserver.cc:460] Webserver started at http://127.1.252.254:45885/ using document root <none> and password file <none>
I20260928 15:01:51.888428 21198 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:51.888473 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:51.888625 21198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:51.889662 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "c4acafd6575b4d649e51524edb457b1a"
format_stamp: "Formatted at 2026-09-28 15:01:51 on dist-test-slave-sf70"
server_key: "e8dd265c01e3afdc5ea22b14bde3b4da"
server_key_iv: "d85333ec557ba3815622dcef74f49f70"
server_key_version: "encryptionkey@0"
I20260928 15:01:51.890025 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "c4acafd6575b4d649e51524edb457b1a"
format_stamp: "Formatted at 2026-09-28 15:01:51 on dist-test-slave-sf70"
server_key: "e8dd265c01e3afdc5ea22b14bde3b4da"
server_key_iv: "d85333ec557ba3815622dcef74f49f70"
server_key_version: "encryptionkey@0"
I20260928 15:01:51.891371 21198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:01:51.892194 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:51.892375 21198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:51.892482 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "c4acafd6575b4d649e51524edb457b1a"
format_stamp: "Formatted at 2026-09-28 15:01:51 on dist-test-slave-sf70"
server_key: "e8dd265c01e3afdc5ea22b14bde3b4da"
server_key_iv: "d85333ec557ba3815622dcef74f49f70"
server_key_version: "encryptionkey@0"
I20260928 15:01:51.892565 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:51.901692 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:51.903565 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:51.903723 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:51.907498 21198 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:42477
I20260928 15:01:51.907552 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:42477 every 8 connection(s)
I20260928 15:01:51.907963 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:51.908774 21267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:51.909538 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21198
I20260928 15:01:51.909667 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:51.909844 2035 external_mini_cluster.cc:1428] Setting key c2f70c762bc985f67488013e97c99ef0
I20260928 15:01:51.912065 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: Bootstrap starting.
I20260928 15:01:51.912647 21267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:51.912940 21267 log.cc:824] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:51.913656 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: No bootstrap required, opened a new log
I20260928 15:01:51.915632 21267 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4acafd6575b4d649e51524edb457b1a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42477 } }
I20260928 15:01:51.915791 21267 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:51.915871 21267 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4acafd6575b4d649e51524edb457b1a, State: Initialized, Role: FOLLOWER
I20260928 15:01:51.916055 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4acafd6575b4d649e51524edb457b1a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42477 } }
I20260928 15:01:51.916153 21267 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 28 15:01:51 dist-test-slave-sf70 krb5kdc[21185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:51.916198 21267 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:51.916241 21267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:51.916989 21267 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4acafd6575b4d649e51524edb457b1a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42477 } }
I20260928 15:01:51.917106 21267 leader_election.cc:302] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4acafd6575b4d649e51524edb457b1a; no voters:
I20260928 15:01:51.917351 21267 leader_election.cc:288] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:51.917384 21272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:51.917500 21272 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [term 1 LEADER]: Becoming Leader. State: Replica: c4acafd6575b4d649e51524edb457b1a, State: Running, Role: LEADER
I20260928 15:01:51.917632 21267 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:51.917892 21272 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4acafd6575b4d649e51524edb457b1a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42477 } }
I20260928 15:01:51.919418 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4acafd6575b4d649e51524edb457b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4acafd6575b4d649e51524edb457b1a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42477 } } }
I20260928 15:01:51.919509 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [sys.catalog]: This master's current role is: LEADER
W20260928 15:01:51.919632 21287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:01:51.919690 21287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:01:51.919687 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4acafd6575b4d649e51524edb457b1a. Latest consensus state: current_term: 1 leader_uuid: "c4acafd6575b4d649e51524edb457b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4acafd6575b4d649e51524edb457b1a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 42477 } } }
I20260928 15:01:51.919843 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:51.919862 21289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:51.920073 21289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:51.920588 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:51.910361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.1.252.254:42477)
0928 15:01:51.910535 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:51.910543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:51.910587 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:01:51.911033 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:51.911039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:51.911088 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:51.911199 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:51.912029 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:51.912980 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:51.913743 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:51.914138 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:51.914257 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:51.916767 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:51.916770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:51.916770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:51.916780 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:51.918763 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:51.919224 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:51.919226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:51.919227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:51.919398 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:51.919598 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:51.919599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:51.919599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:51.919994 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:51.920165 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:51.920280 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 15:01:51.920415 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 15:01:51.921633 21289 catalog_manager.cc:1347] Generated new cluster ID: 09300d595b5b418d955acde47abf5e48
I20260928 15:01:51.921684 21289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:51.935465 21289 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:51.936062 21289 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:51.942633 21289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: Generated new TSK 0
I20260928 15:01:51.942804 21289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.252.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
Entry for principal kudu/127.1.252.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab.
I20260928 15:01:51.971336 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42477
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:52.050755 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.050941 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.050963 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.052155 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.052217 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.052244 21293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:52.053617 21293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.193.keytab
--principal=kudu/127.1.252.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:52.053793 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.053979 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.059435 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.059514 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.059636 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.059615 21293 server_base.cc:1034] running on GCE node
I20260928 15:01:52.059831 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.060104 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.061260 21293 hybrid_clock.cc:648] HybridClock initialized: now 1790607712061242 us; error 33 us; skew 500 ppm
Sep 28 15:01:52 dist-test-slave-sf70 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:52.063807 21293 init.cc:377] Logged in from keytab as kudu/127.1.252.193@KRBTEST.COM (short username kudu)
I20260928 15:01:52.064248 21293 webserver.cc:460] Webserver started at http://127.1.252.193:34601/ using document root <none> and password file <none>
I20260928 15:01:52.064491 21293 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.064559 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.064723 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.065797 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "36165b71457843ba98cc1755fd9c5b63"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "36bd3e604689b56372fdc3ba6eed75ae"
server_key_iv: "44fa0b19755ba3b2d46b70353098d757"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.066156 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "36165b71457843ba98cc1755fd9c5b63"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "36bd3e604689b56372fdc3ba6eed75ae"
server_key_iv: "44fa0b19755ba3b2d46b70353098d757"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.067509 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:52.068318 21308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.068547 21293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:01:52.068645 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "36165b71457843ba98cc1755fd9c5b63"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "36bd3e604689b56372fdc3ba6eed75ae"
server_key_iv: "44fa0b19755ba3b2d46b70353098d757"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.068727 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.094847 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.097240 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.097406 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.097674 21293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:52.097990 21293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:52.098044 21293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.098078 21293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:52.098109 21293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.104123 21293 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:40721
I20260928 15:01:52.104187 21422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:40721 every 8 connection(s)
I20260928 15:01:52.104542 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:52.106240 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21293
I20260928 15:01:52.106372 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:52.106600 2035 external_mini_cluster.cc:1428] Setting key 1c97144a6ca39f4958d7e99044c75f84
Sep 28 15:01:52 dist-test-slave-sf70 krb5kdc[21185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.193@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:52.115419 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.105280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42477 (local address 127.1.252.193:43131)
0928 15:01:52.105425 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.105466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.105942 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.106202 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.106207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.106966 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:52.107231 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.107236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.108220 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.108221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.109097 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.109099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.109649 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.109650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.109764 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.110276 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:52.110287 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:52.112061 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:52.114058 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:52.114065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:52.114067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:52.114275 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:52.114497 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:52.114498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:52.114498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:52.114635 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:52.114957 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:52.114962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:52.115072 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.115152 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.115266 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260928 15:01:52.115439 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.105313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43131 (local address 127.1.252.254:42477)
0928 15:01:52.105383 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.105385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.105578 (+ 193us) server_negotiation.cc:407] Connection header received
0928 15:01:52.106045 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.106046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.106056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.106110 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.107338 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.108109 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.109185 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.109520 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.109637 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.112155 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.112157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.112157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:52.112166 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.113972 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.114355 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.114356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.114356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.114424 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.114716 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.114717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.114717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.114880 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.114964 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.115207 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.115316 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 15:01:52.116281 21423 heartbeater.cc:347] Connected to a master server at 127.1.252.254:42477
I20260928 15:01:52.116384 21423 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.252.194@KRBTEST.COM; defaulting to no policy
I20260928 15:01:52.116608 21423 heartbeater.cc:511] Master 127.1.252.254:42477 requested a full tablet report, sending...
I20260928 15:01:52.117141 21230 ts_manager.cc:194] Registered new tserver with Master: 36165b71457843ba98cc1755fd9c5b63 (127.1.252.193:40721)
I20260928 15:01:52.117893 21230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.193@KRBTEST.COM'} at 127.1.252.193:43131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
Entry for principal kudu/127.1.252.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab.
I20260928 15:01:52.135756 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42477
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:52.216075 21428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.216300 21428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.216339 21428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.217501 21428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.217566 21428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.217613 21428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:52.219303 21428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.194.keytab
--principal=kudu/127.1.252.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:52.219558 21428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.219801 21428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.224892 21433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.224960 21434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.224850 21436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.225096 21428 server_base.cc:1034] running on GCE node
I20260928 15:01:52.225266 21428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.225466 21428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.226619 21428 hybrid_clock.cc:648] HybridClock initialized: now 1790607712226621 us; error 45 us; skew 500 ppm
Sep 28 15:01:52 dist-test-slave-sf70 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:52.229169 21428 init.cc:377] Logged in from keytab as kudu/127.1.252.194@KRBTEST.COM (short username kudu)
I20260928 15:01:52.229549 21428 webserver.cc:460] Webserver started at http://127.1.252.194:45573/ using document root <none> and password file <none>
I20260928 15:01:52.229776 21428 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.229847 21428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.230014 21428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.231035 21428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "556775ff21594f55902b53b5b74e3857"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "3d8bbc53dd3d4a4e35436c1f57903e97"
server_key_iv: "142996fb99c21d7190f1e1e0d742e40d"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.231393 21428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "556775ff21594f55902b53b5b74e3857"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "3d8bbc53dd3d4a4e35436c1f57903e97"
server_key_iv: "142996fb99c21d7190f1e1e0d742e40d"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.232798 21428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:52.233632 21443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.233870 21428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:52.233979 21428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "556775ff21594f55902b53b5b74e3857"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "3d8bbc53dd3d4a4e35436c1f57903e97"
server_key_iv: "142996fb99c21d7190f1e1e0d742e40d"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.234091 21428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.246029 21428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.247876 21428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.248109 21428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.248342 21428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:52.248643 21428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:52.248677 21428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.248725 21428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:52.248751 21428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.254910 21428 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:39047
I20260928 15:01:52.254957 21557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:39047 every 8 connection(s)
I20260928 15:01:52.255257 21428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:52.260449 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21428
I20260928 15:01:52.260569 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:52.260756 2035 external_mini_cluster.cc:1428] Setting key 17a19679f71760641f6946357dba14bd
Sep 28 15:01:52 dist-test-slave-sf70 krb5kdc[21185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.194@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:52.264477 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.256007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:57721 (local address 127.1.252.254:42477)
0928 15:01:52.256067 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.256070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.256268 (+ 198us) server_negotiation.cc:407] Connection header received
0928 15:01:52.256775 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.256776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.256788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.256838 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.257735 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.258291 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.259231 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.259437 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.259494 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.261725 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.261726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.261726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:52.261733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.263081 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.263480 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.263482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.263482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.263549 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.263796 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.263797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.263797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.263962 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.264043 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.264308 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.264388 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:01:52.264689 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.256008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42477 (local address 127.1.252.194:57721)
0928 15:01:52.256169 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.256203 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.256692 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.256889 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.256892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.257429 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:52.257628 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.257632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.258373 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.258373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.259153 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.259155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.259510 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.259511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.259630 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.260126 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:52.260134 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:52.261615 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:52.263179 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:52.263186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:52.263188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:52.263405 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:52.263623 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:52.263623 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:52.263624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:52.263730 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:52.264035 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:52.264049 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:52.264180 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.264288 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.264439 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260928 15:01:52.265199 21558 heartbeater.cc:347] Connected to a master server at 127.1.252.254:42477
I20260928 15:01:52.265290 21558 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:52.265475 21558 heartbeater.cc:511] Master 127.1.252.254:42477 requested a full tablet report, sending...
I20260928 15:01:52.265902 21230 ts_manager.cc:194] Registered new tserver with Master: 556775ff21594f55902b53b5b74e3857 (127.1.252.194:39047)
I20260928 15:01:52.266288 21230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.194@KRBTEST.COM'} at 127.1.252.194:57721
WARNING: no policy specified for kudu/127.1.252.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
Entry for principal kudu/127.1.252.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab.
I20260928 15:01:52.286943 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42477
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:52.366984 21563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.367184 21563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.367206 21563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.368360 21563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.368400 21563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.368426 21563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:52.369889 21563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.195.keytab
--principal=kudu/127.1.252.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:42477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:52.370129 21563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.370368 21563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.375527 21568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.375590 21569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.375571 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.375823 21563 server_base.cc:1034] running on GCE node
I20260928 15:01:52.376168 21563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.376407 21563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.377630 21563 hybrid_clock.cc:648] HybridClock initialized: now 1790607712377621 us; error 32 us; skew 500 ppm
Sep 28 15:01:52 dist-test-slave-sf70 krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:52.380383 21563 init.cc:377] Logged in from keytab as kudu/127.1.252.195@KRBTEST.COM (short username kudu)
I20260928 15:01:52.380828 21563 webserver.cc:460] Webserver started at http://127.1.252.195:39817/ using document root <none> and password file <none>
I20260928 15:01:52.381090 21563 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.381137 21563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.381307 21563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.382424 21563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "d22d802682e04cc4af37e6fb26198859"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "6c0ab9d4228232647df7c298ce41a74a"
server_key_iv: "a187d5b782cc002bedf9e3a207b07019"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.382827 21563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "d22d802682e04cc4af37e6fb26198859"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "6c0ab9d4228232647df7c298ce41a74a"
server_key_iv: "a187d5b782cc002bedf9e3a207b07019"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.384243 21563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:01:52.385035 21578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.385257 21563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:01:52.385353 21563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "d22d802682e04cc4af37e6fb26198859"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "6c0ab9d4228232647df7c298ce41a74a"
server_key_iv: "a187d5b782cc002bedf9e3a207b07019"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.385432 21563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.395146 21563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.396987 21563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.397140 21563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.397346 21563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:52.397646 21563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:52.397699 21563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.397739 21563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:52.397768 21563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.403258 21563 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:37885
I20260928 15:01:52.403340 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:37885 every 8 connection(s)
I20260928 15:01:52.403640 21563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:01:52 dist-test-slave-sf70 krb5kdc[21185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607712, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.195@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:01:52.411507 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21563
I20260928 15:01:52.411631 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:52.411794 2035 external_mini_cluster.cc:1428] Setting key 462093fe08a8184e57dde8b2e46b8d60
I20260928 15:01:52.412750 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.404327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:32783 (local address 127.1.252.254:42477)
0928 15:01:52.404367 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.404370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.404594 (+ 224us) server_negotiation.cc:407] Connection header received
0928 15:01:52.405101 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.405102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.405112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.405166 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.406039 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.406574 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.407531 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.407721 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.407785 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.409903 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.409904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.409905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:52.409911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.411131 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.411524 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.411526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.411526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.411597 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.412003 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.412004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.412005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.412181 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.412261 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.412607 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.412687 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:01:52.413015 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.404303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:42477 (local address 127.1.252.195:32783)
0928 15:01:52.404463 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.404502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.404994 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.405215 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.405219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.405760 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:52.405941 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.405945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.406661 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.406662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.407423 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.407425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.407802 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.407803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.407950 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.408389 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:01:52.408397 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:52.409769 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:52.411202 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:52.411208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:52.411210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:52.411441 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:52.411751 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:01:52.411752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:01:52.411752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:01:52.411922 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:01:52.412265 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:52.412273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:52.412453 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.412595 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.412773 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 15:01:52.413480 21692 heartbeater.cc:347] Connected to a master server at 127.1.252.254:42477
I20260928 15:01:52.413584 21692 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:52.413757 21692 heartbeater.cc:511] Master 127.1.252.254:42477 requested a full tablet report, sending...
I20260928 15:01:52.414122 21230 ts_manager.cc:194] Registered new tserver with Master: d22d802682e04cc4af37e6fb26198859 (127.1.252.195:37885)
I20260928 15:01:52.414487 21230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.252.195@KRBTEST.COM'} at 127.1.252.195:32783
I20260928 15:01:52.423632 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:52.430850 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.425039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.1.252.254:42477)
0928 15:01:52.425102 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.425104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.425211 (+ 107us) server_negotiation.cc:407] Connection header received
0928 15:01:52.425263 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.425263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.425272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.425311 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.425797 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.426261 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.427035 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.427216 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.427269 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.428487 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.428488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.428488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:52.428493 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.429833 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.430158 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.430159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.430159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.430213 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:52.430384 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:52.430384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:52.430385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:52.430546 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.430624 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.430731 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.430791 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:01:52.432386 21230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:52.433943 21230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:52.441035 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.436336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40721 (local address 127.0.0.1:56316)
0928 15:01:52.436867 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.436877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.436925 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.437185 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.437188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.437192 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:52.437309 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.437311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.438619 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.438620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.439704 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.439706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.440663 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.440664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.440766 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.440781 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.440864 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.440923 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":64,"thread_start_us":54,"threads_started":1}
I20260928 15:01:52.441035 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.436725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.1.252.193:40721)
0928 15:01:52.437004 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.437008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.437019 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:01:52.437063 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.437065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.437081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.437169 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:52.437413 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.438503 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.439796 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.440560 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.440636 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.440691 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.440897 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.440948 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":122,"threads_started":1}
I20260928 15:01:52.441200 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.436315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:39047 (local address 127.0.0.1:59998)
0928 15:01:52.436433 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.436513 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.436580 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.437185 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.437188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.437190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:52.437308 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.437311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.438753 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.438754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.440038 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.440041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.440951 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.440952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.441026 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.441030 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.441063 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.441101 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":130,"thread_start_us":68,"threads_started":1}
I20260928 15:01:52.441429 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.436540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:37885 (local address 127.0.0.1:57486)
0928 15:01:52.437267 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.437276 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.437354 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.437533 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.437534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.437535 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:52.437590 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.437592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.438611 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.438612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.440105 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.440107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.441185 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.441186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.441243 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.441247 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.441299 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.441342 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":69,"thread_start_us":64,"threads_started":1}
I20260928 15:01:52.441529 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.436513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.1.252.194:39047)
0928 15:01:52.437021 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.437026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.437035 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:01:52.437070 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.437071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.437086 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.437169 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:52.437404 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.438661 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.440130 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.440818 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.441317 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.441385 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.441411 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.441454 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":110,"threads_started":1}
I20260928 15:01:52.441557 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.436654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.1.252.195:37885)
0928 15:01:52.436919 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.436925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.437350 (+ 425us) server_negotiation.cc:407] Connection header received
0928 15:01:52.437425 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.437427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.437452 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.437537 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:52.437646 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.438503 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.440172 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.441010 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.441076 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.441142 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.441442 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.441482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260928 15:01:52.442485 21491 tablet_service.cc:1467] Processing CreateTablet for tablet ae7a9490e5774c8090c6f2035c94a924 (DEFAULT_TABLE table=test-table [id=a79747592a0543418e4b9b8de3f91cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:52.442464 21626 tablet_service.cc:1467] Processing CreateTablet for tablet ae7a9490e5774c8090c6f2035c94a924 (DEFAULT_TABLE table=test-table [id=a79747592a0543418e4b9b8de3f91cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:52.442669 21356 tablet_service.cc:1467] Processing CreateTablet for tablet ae7a9490e5774c8090c6f2035c94a924 (DEFAULT_TABLE table=test-table [id=a79747592a0543418e4b9b8de3f91cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:52.442848 21626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7a9490e5774c8090c6f2035c94a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:52.442862 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7a9490e5774c8090c6f2035c94a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:52.442952 21356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7a9490e5774c8090c6f2035c94a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:52.445571 21706 tablet_bootstrap.cc:492] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Bootstrap starting.
I20260928 15:01:52.446273 21706 tablet_bootstrap.cc:654] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:52.446139 21707 tablet_bootstrap.cc:492] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: Bootstrap starting.
I20260928 15:01:52.446894 21706 log.cc:824] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:52.447003 21707 tablet_bootstrap.cc:654] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:52.447588 21707 log.cc:824] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:52.447928 21706 tablet_bootstrap.cc:492] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: No bootstrap required, opened a new log
I20260928 15:01:52.447970 21708 tablet_bootstrap.cc:492] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Bootstrap starting.
I20260928 15:01:52.448024 21706 ts_tablet_manager.cc:1397] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:01:52.448348 21707 tablet_bootstrap.cc:492] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: No bootstrap required, opened a new log
I20260928 15:01:52.448432 21707 ts_tablet_manager.cc:1397] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:52.448622 21708 tablet_bootstrap.cc:654] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:52.449195 21708 log.cc:824] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:52.449941 21708 tablet_bootstrap.cc:492] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: No bootstrap required, opened a new log
I20260928 15:01:52.450012 21708 ts_tablet_manager.cc:1397] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:01:52.450176 21706 raft_consensus.cc:348] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.450227 21707 raft_consensus.cc:348] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.450347 21706 raft_consensus.cc:374] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:52.450374 21707 raft_consensus.cc:374] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:52.450381 21706 raft_consensus.cc:729] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556775ff21594f55902b53b5b74e3857, State: Initialized, Role: FOLLOWER
I20260928 15:01:52.450417 21707 raft_consensus.cc:729] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36165b71457843ba98cc1755fd9c5b63, State: Initialized, Role: FOLLOWER
I20260928 15:01:52.450528 21706 consensus_queue.cc:260] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.450541 21707 consensus_queue.cc:260] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.450793 21706 ts_tablet_manager.cc:1428] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:52.450835 21707 ts_tablet_manager.cc:1428] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:01:52.450928 21558 heartbeater.cc:503] Master 127.1.252.254:42477 was elected leader, sending a full tablet report...
I20260928 15:01:52.450969 21423 heartbeater.cc:503] Master 127.1.252.254:42477 was elected leader, sending a full tablet report...
I20260928 15:01:52.452265 21708 raft_consensus.cc:348] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.452386 21708 raft_consensus.cc:374] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:52.452414 21708 raft_consensus.cc:729] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22d802682e04cc4af37e6fb26198859, State: Initialized, Role: FOLLOWER
I20260928 15:01:52.452543 21708 consensus_queue.cc:260] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.452785 21708 ts_tablet_manager.cc:1428] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:01:52.452840 21692 heartbeater.cc:503] Master 127.1.252.254:42477 was elected leader, sending a full tablet report...
I20260928 15:01:52.484871 21712 raft_consensus.cc:482] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:52.484995 21712 raft_consensus.cc:504] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.485357 21712 leader_election.cc:288] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d22d802682e04cc4af37e6fb26198859 (127.1.252.195:37885), 36165b71457843ba98cc1755fd9c5b63 (127.1.252.193:40721)
I20260928 15:01:52.488265 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.485612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:57809 (local address 127.1.252.195:37885)
0928 15:01:52.485671 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.485673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.485683 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:01:52.485713 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.485714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.485725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.485806 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:52.485975 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.486467 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.487491 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.487873 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.487976 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.487986 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.488153 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.488192 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:01:52.488266 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.485505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:37885 (local address 127.1.252.194:57809)
0928 15:01:52.485558 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.485570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.485619 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.485812 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.485813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.485815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:52.485902 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.485904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.486569 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.486570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.487398 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.487400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.487974 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.487975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.488075 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.488078 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.488146 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.488193 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":50}
I20260928 15:01:52.488344 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.485670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:57345 (local address 127.1.252.193:40721)
0928 15:01:52.485725 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.485727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.485794 (+ 67us) server_negotiation.cc:407] Connection header received
0928 15:01:52.485844 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.485845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.485854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.485927 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:52.486121 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.486608 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.487535 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.487983 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.488066 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.488074 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.488239 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.488275 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 15:01:52.488386 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.485550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40721 (local address 127.1.252.194:57345)
0928 15:01:52.485725 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.485734 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.485786 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.485931 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.485932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.485934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:52.486014 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.486016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.486704 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.486705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.487442 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.487444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.488069 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.488070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.488170 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.488181 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.488234 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.488273 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":116,"threads_started":1}
I20260928 15:01:52.488662 21646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7a9490e5774c8090c6f2035c94a924" candidate_uuid: "556775ff21594f55902b53b5b74e3857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22d802682e04cc4af37e6fb26198859" is_pre_election: true
I20260928 15:01:52.488874 21646 raft_consensus.cc:2454] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556775ff21594f55902b53b5b74e3857 in term 0.
I20260928 15:01:52.488948 21376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7a9490e5774c8090c6f2035c94a924" candidate_uuid: "556775ff21594f55902b53b5b74e3857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36165b71457843ba98cc1755fd9c5b63" is_pre_election: true
I20260928 15:01:52.489120 21376 raft_consensus.cc:2454] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556775ff21594f55902b53b5b74e3857 in term 0.
I20260928 15:01:52.489364 21446 leader_election.cc:302] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556775ff21594f55902b53b5b74e3857, d22d802682e04cc4af37e6fb26198859; no voters:
I20260928 15:01:52.489507 21712 raft_consensus.cc:2781] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:52.489564 21712 raft_consensus.cc:482] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:52.489588 21712 raft_consensus.cc:3037] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:52.490298 21712 raft_consensus.cc:504] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.490463 21712 leader_election.cc:288] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [CANDIDATE]: Term 1 election: Requested vote from peers d22d802682e04cc4af37e6fb26198859 (127.1.252.195:37885), 36165b71457843ba98cc1755fd9c5b63 (127.1.252.193:40721)
I20260928 15:01:52.490664 21646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7a9490e5774c8090c6f2035c94a924" candidate_uuid: "556775ff21594f55902b53b5b74e3857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22d802682e04cc4af37e6fb26198859"
I20260928 15:01:52.490664 21376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7a9490e5774c8090c6f2035c94a924" candidate_uuid: "556775ff21594f55902b53b5b74e3857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36165b71457843ba98cc1755fd9c5b63"
I20260928 15:01:52.490739 21646 raft_consensus.cc:3037] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:52.490739 21376 raft_consensus.cc:3037] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:52.491611 21646 raft_consensus.cc:2454] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556775ff21594f55902b53b5b74e3857 in term 1.
I20260928 15:01:52.491611 21376 raft_consensus.cc:2454] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556775ff21594f55902b53b5b74e3857 in term 1.
I20260928 15:01:52.491777 21446 leader_election.cc:302] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556775ff21594f55902b53b5b74e3857, d22d802682e04cc4af37e6fb26198859; no voters:
I20260928 15:01:52.491925 21712 raft_consensus.cc:2781] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:52.492115 21712 raft_consensus.cc:686] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 1 LEADER]: Becoming Leader. State: Replica: 556775ff21594f55902b53b5b74e3857, State: Running, Role: LEADER
I20260928 15:01:52.492215 21712 consensus_queue.cc:237] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } }
I20260928 15:01:52.493081 21230 catalog_manager.cc:5495] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 reported cstate change: term changed from 0 to 1, leader changed from <none> to 556775ff21594f55902b53b5b74e3857 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "556775ff21594f55902b53b5b74e3857" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556775ff21594f55902b53b5b74e3857" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260928 15:01:52.497980 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.495844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.1.252.194:39047)
0928 15:01:52.495927 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.495929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.495937 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:01:52.496008 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.496009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.496023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.496096 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:52.496226 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.496656 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.497362 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.497532 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.497580 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.497711 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:52.497795 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:52.497851 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.497882 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.497899 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 15:01:52.499550 21376 raft_consensus.cc:1261] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 1 FOLLOWER]: Refusing update from remote peer 556775ff21594f55902b53b5b74e3857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:52.499615 21646 raft_consensus.cc:1261] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 1 FOLLOWER]: Refusing update from remote peer 556775ff21594f55902b53b5b74e3857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:52.499966 21716 consensus_queue.cc:1035] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22d802682e04cc4af37e6fb26198859" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 37885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:52.500111 21712 consensus_queue.cc:1035] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36165b71457843ba98cc1755fd9c5b63" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:52.502642 21720 mvcc.cc:205] Tried to move back new op lower bound from 7334329190395940864 to 7334329190368940032. Current Snapshot: MvccSnapshot[applied={T|T < 7334329190395940864}]
I20260928 15:01:52.504769 21229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:52.504865 21229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:52.505581 21229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4acafd6575b4d649e51524edb457b1a: Sending DeleteTablet for 3 replicas of tablet ae7a9490e5774c8090c6f2035c94a924
I20260928 15:01:52.505977 21356 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7a9490e5774c8090c6f2035c94a924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:52 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:56316
I20260928 15:01:52.506053 21491 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7a9490e5774c8090c6f2035c94a924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:52 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:59998
I20260928 15:01:52.506062 21626 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7a9490e5774c8090c6f2035c94a924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:52 UTC) from {username='kudu', principal='kudu/127.1.252.254@KRBTEST.COM'} at 127.0.0.1:57486
I20260928 15:01:52.506258 21730 tablet_replica.cc:330] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: stopping tablet replica
I20260928 15:01:52.506335 21729 tablet_replica.cc:330] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63: stopping tablet replica
I20260928 15:01:52.506349 21730 raft_consensus.cc:2229] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:52.506434 21730 raft_consensus.cc:2258] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:52.506431 21729 raft_consensus.cc:2229] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:52.506528 21729 raft_consensus.cc:2258] T ae7a9490e5774c8090c6f2035c94a924 P 36165b71457843ba98cc1755fd9c5b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:52.506538 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21293
I20260928 15:01:52.506590 21731 tablet_replica.cc:330] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: stopping tablet replica
I20260928 15:01:52.507354 21731 raft_consensus.cc:2229] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:52.507473 21731 raft_consensus.cc:2258] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:52.507763 21730 ts_tablet_manager.cc:1905] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:52.509282 21731 ts_tablet_manager.cc:1905] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:52.510497 21730 ts_tablet_manager.cc:1918] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:52.510593 21730 log.cc:1196] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-1/wal/wals/ae7a9490e5774c8090c6f2035c94a924
I20260928 15:01:52.510857 21730 ts_tablet_manager.cc:1939] T ae7a9490e5774c8090c6f2035c94a924 P 556775ff21594f55902b53b5b74e3857: Deleting consensus metadata
I20260928 15:01:52.511266 21731 ts_tablet_manager.cc:1918] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:52.511293 21216 catalog_manager.cc:4841] TS 556775ff21594f55902b53b5b74e3857 (127.1.252.194:39047): tablet ae7a9490e5774c8090c6f2035c94a924 (table test-table [id=a79747592a0543418e4b9b8de3f91cc1]) successfully deleted
I20260928 15:01:52.511332 21731 log.cc:1196] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/ae7a9490e5774c8090c6f2035c94a924
I20260928 15:01:52.511591 21731 ts_tablet_manager.cc:1939] T ae7a9490e5774c8090c6f2035c94a924 P d22d802682e04cc4af37e6fb26198859: Deleting consensus metadata
I20260928 15:01:52.511981 21216 catalog_manager.cc:4841] TS d22d802682e04cc4af37e6fb26198859 (127.1.252.195:37885): tablet ae7a9490e5774c8090c6f2035c94a924 (table test-table [id=a79747592a0543418e4b9b8de3f91cc1]) successfully deleted
W20260928 15:01:52.513136 21215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:40721 (error 108)
I20260928 15:01:52.513617 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.513485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:40721 (local address 127.0.0.1:56332)
0928 15:01:52.513539 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.513583 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 15:01:52.513672 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21428
W20260928 15:01:52.513756 21215 catalog_manager.cc:4584] TS 36165b71457843ba98cc1755fd9c5b63 (127.1.252.193:40721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae7a9490e5774c8090c6f2035c94a924: Network error: Client connection negotiation failed: client connection to 127.1.252.193:40721: connect: Connection refused (error 111)
I20260928 15:01:52.518599 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21563
I20260928 15:01:52.523518 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21198
2026-09-28T15:01:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1892 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T15:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:52Z Disabled control of system clock
I20260928 15:01:52.551407 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35057
--webserver_interface=127.1.252.254
--webserver_port=0
--builtin_ntp_servers=127.1.252.212:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:52.632179 21737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.632411 21737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.632439 21737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.633549 21737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:52.633585 21737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.633598 21737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.633610 21737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:52.633622 21737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:52.634863 21737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:35057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:52.635054 21737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.635362 21737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.640187 21742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.640215 21745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.640187 21743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.640761 21737 server_base.cc:1034] running on GCE node
I20260928 15:01:52.641022 21737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.641314 21737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.642480 21737 hybrid_clock.cc:648] HybridClock initialized: now 1790607712642462 us; error 37 us; skew 500 ppm
I20260928 15:01:52.643818 21737 webserver.cc:460] Webserver started at http://127.1.252.254:41291/ using document root <none> and password file <none>
I20260928 15:01:52.644083 21737 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.644153 21737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.644299 21737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.645392 21737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "88606b9879624c6b9e1c8720e15741cc"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "0d20f10b8966f0442f7eb09a307a4611"
server_key_iv: "3a80efd1b6a7a5b9be1ce18312466e7c"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.645756 21737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "88606b9879624c6b9e1c8720e15741cc"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "0d20f10b8966f0442f7eb09a307a4611"
server_key_iv: "3a80efd1b6a7a5b9be1ce18312466e7c"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.647049 21737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:52.647806 21751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.648025 21737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:52.648124 21737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "88606b9879624c6b9e1c8720e15741cc"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "0d20f10b8966f0442f7eb09a307a4611"
server_key_iv: "3a80efd1b6a7a5b9be1ce18312466e7c"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.648216 21737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.660713 21737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.661026 21737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.661183 21737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.665413 21737 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:35057
I20260928 15:01:52.665452 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:35057 every 8 connection(s)
I20260928 15:01:52.665825 21737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:52.666844 21804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:52.669277 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc: Bootstrap starting.
I20260928 15:01:52.669865 21804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:52.670192 21804 log.cc:824] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:52.670964 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc: No bootstrap required, opened a new log
I20260928 15:01:52.672802 21804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88606b9879624c6b9e1c8720e15741cc" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35057 } }
I20260928 15:01:52.672952 21804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:52.672977 21804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88606b9879624c6b9e1c8720e15741cc, State: Initialized, Role: FOLLOWER
I20260928 15:01:52.673136 21804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88606b9879624c6b9e1c8720e15741cc" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35057 } }
I20260928 15:01:52.673202 21804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:52.673223 21804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:52.673270 21804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:52.673781 21804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88606b9879624c6b9e1c8720e15741cc" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35057 } }
I20260928 15:01:52.673866 21804 leader_election.cc:302] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88606b9879624c6b9e1c8720e15741cc; no voters:
I20260928 15:01:52.674086 21804 leader_election.cc:288] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:52.674142 21807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:52.674358 21807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [term 1 LEADER]: Becoming Leader. State: Replica: 88606b9879624c6b9e1c8720e15741cc, State: Running, Role: LEADER
I20260928 15:01:52.674381 21804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:52.674683 21807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88606b9879624c6b9e1c8720e15741cc" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35057 } }
I20260928 15:01:52.675117 21809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88606b9879624c6b9e1c8720e15741cc. Latest consensus state: current_term: 1 leader_uuid: "88606b9879624c6b9e1c8720e15741cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88606b9879624c6b9e1c8720e15741cc" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35057 } } }
I20260928 15:01:52.675201 21808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88606b9879624c6b9e1c8720e15741cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88606b9879624c6b9e1c8720e15741cc" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35057 } } }
I20260928 15:01:52.675225 21809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:52.675271 21808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:52.675540 21816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:52.675925 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21737
I20260928 15:01:52.676035 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:52.676124 21816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:52.676246 2035 external_mini_cluster.cc:1428] Setting key 270adb21a34cda6e05549ab01a506c3b
I20260928 15:01:52.680333 21816 catalog_manager.cc:1347] Generated new cluster ID: 1a4888bf8bf444e2987368e4cfcd3de3
I20260928 15:01:52.680392 21816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:52.682402 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.676950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.1.252.254:35057)
0928 15:01:52.677162 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.677171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.677205 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:01:52.677319 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.677327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.677409 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.677561 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.677788 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.679162 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.680008 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.680674 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.680802 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.681546 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.681549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.681550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:52.681762 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.681806 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:52.681821 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.681896 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.681969 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.682171 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260928 15:01:52.686962 21816 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:52.687422 21816 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:52.691192 21816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc: Generated new TSK 0
I20260928 15:01:52.691372 21816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:52.693225 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.193:0
--local_ip_for_outbound_sockets=127.1.252.193
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35057
--builtin_ntp_servers=127.1.252.212:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:52.775844 21828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.776101 21828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.776137 21828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.777293 21828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.777360 21828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.777410 21828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.193
I20260928 15:01:52.778774 21828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.252.193
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.193
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:52.779043 21828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.779359 21828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.785153 21833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.785213 21836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.785879 21828 server_base.cc:1034] running on GCE node
W20260928 15:01:52.786103 21834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.786343 21828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.786641 21828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.787823 21828 hybrid_clock.cc:648] HybridClock initialized: now 1790607712787807 us; error 30 us; skew 500 ppm
I20260928 15:01:52.789314 21828 webserver.cc:460] Webserver started at http://127.1.252.193:36345/ using document root <none> and password file <none>
I20260928 15:01:52.789573 21828 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.789629 21828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.789743 21828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.791061 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data/instance:
uuid: "500798626d92407caf58851b62b4724b"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "869ae7b91e5eeaa773c315f9cacc551b"
server_key_iv: "bcca0d0b2f344c03ff1ef9d86cce4013"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.791441 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance:
uuid: "500798626d92407caf58851b62b4724b"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "869ae7b91e5eeaa773c315f9cacc551b"
server_key_iv: "bcca0d0b2f344c03ff1ef9d86cce4013"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.793097 21828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:52.794057 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.794297 21828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:52.794394 21828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal
uuid: "500798626d92407caf58851b62b4724b"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "869ae7b91e5eeaa773c315f9cacc551b"
server_key_iv: "bcca0d0b2f344c03ff1ef9d86cce4013"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.794466 21828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.816324 21828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.816623 21828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.816735 21828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.816928 21828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:52.817291 21828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:52.817327 21828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.817349 21828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:52.817361 21828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.823283 21828 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.193:34109
I20260928 15:01:52.823328 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.193:34109 every 8 connection(s)
I20260928 15:01:52.823663 21828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/data/info.pb
I20260928 15:01:52.828799 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21828
I20260928 15:01:52.828912 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-0/wal/instance
I20260928 15:01:52.829041 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.824440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.193:43609 (local address 127.1.252.254:35057)
0928 15:01:52.824486 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.824488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.824844 (+ 356us) server_negotiation.cc:407] Connection header received
0928 15:01:52.825274 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.825275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.825289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.825345 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.826009 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.826515 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.827469 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.827671 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.827734 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.828610 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.828611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.828611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:52.828616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.828664 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:52.828669 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.828726 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.828856 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.828930 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:01:52.829126 2035 external_mini_cluster.cc:1428] Setting key acb0cd933474c08d59e93fd3e0e67f31
I20260928 15:01:52.829198 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.824489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35057 (local address 127.1.252.193:43609)
0928 15:01:52.824679 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.824714 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.825184 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.825395 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.825398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.825406 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:52.825912 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.825916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.826599 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.826600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.827369 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.827371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.827752 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.827753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.827861 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.828430 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:52.828431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:52.828459 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:52.828482 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:52.828745 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:52.828753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:52.828756 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.828837 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.828961 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260928 15:01:52.830009 21957 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35057
I20260928 15:01:52.830139 21957 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:52.830156 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.194:0
--local_ip_for_outbound_sockets=127.1.252.194
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35057
--builtin_ntp_servers=127.1.252.212:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:01:52.830350 21957 heartbeater.cc:511] Master 127.1.252.254:35057 requested a full tablet report, sending...
I20260928 15:01:52.830916 21768 ts_manager.cc:194] Registered new tserver with Master: 500798626d92407caf58851b62b4724b (127.1.252.193:34109)
I20260928 15:01:52.831785 21768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.193:43609
W20260928 15:01:52.911437 21960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:52.911731 21960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:52.911768 21960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:52.913306 21960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:52.913388 21960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:52.913429 21960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.194
I20260928 15:01:52.914968 21960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.252.194
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.194
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:52.915256 21960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:52.915572 21960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:52.920720 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.920728 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:52.920737 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:52.920878 21960 server_base.cc:1034] running on GCE node
I20260928 15:01:52.921146 21960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:52.921367 21960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:52.922505 21960 hybrid_clock.cc:648] HybridClock initialized: now 1790607712922480 us; error 44 us; skew 500 ppm
I20260928 15:01:52.923839 21960 webserver.cc:460] Webserver started at http://127.1.252.194:41985/ using document root <none> and password file <none>
I20260928 15:01:52.924121 21960 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:52.924171 21960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:52.924309 21960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:52.925427 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data/instance:
uuid: "4d2e0136566a4958887a5c7ca32341e1"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "0ee220ac5234197ce7807bad74e7f61f"
server_key_iv: "b3660acf826ac3d107d533a79834eabc"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.925792 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance:
uuid: "4d2e0136566a4958887a5c7ca32341e1"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "0ee220ac5234197ce7807bad74e7f61f"
server_key_iv: "b3660acf826ac3d107d533a79834eabc"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.927078 21960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:52.927799 21974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.928054 21960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:52.928158 21960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal
uuid: "4d2e0136566a4958887a5c7ca32341e1"
format_stamp: "Formatted at 2026-09-28 15:01:52 on dist-test-slave-sf70"
server_key: "0ee220ac5234197ce7807bad74e7f61f"
server_key_iv: "b3660acf826ac3d107d533a79834eabc"
server_key_version: "encryptionkey@0"
I20260928 15:01:52.928234 21960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:52.938108 21960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:52.938383 21960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:52.938529 21960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:52.938738 21960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:52.939050 21960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:52.939105 21960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.939145 21960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:52.939172 21960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:52.945317 21960 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.194:39717
I20260928 15:01:52.945346 22088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.194:39717 every 8 connection(s)
I20260928 15:01:52.945686 21960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/data/info.pb
I20260928 15:01:52.950783 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.946412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:53763 (local address 127.1.252.254:35057)
0928 15:01:52.946478 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:52.946481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:52.946746 (+ 265us) server_negotiation.cc:407] Connection header received
0928 15:01:52.947209 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:52.947209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:52.947219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:52.947284 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:52.947946 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.948465 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.949437 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.949642 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.949700 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.950525 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:52.950527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:52.950528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:52.950532 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:52.950549 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:52.950553 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:52.950615 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:52.950702 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 15:01:52.950725 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:01:52.950965 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:52.946402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35057 (local address 127.1.252.194:53763)
0928 15:01:52.946620 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:52.946659 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:52.947124 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:52.947336 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:52.947340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:52.947348 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:52.947828 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.947832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.948575 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.948576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.949355 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:52.949357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:52.949719 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:52.949720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:52.949827 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:52.950365 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:52.950365 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:52.950391 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:52.950413 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:52.950624 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:52.950629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:52.950631 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:52.950686 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 15:01:52.950765 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260928 15:01:52.951508 22089 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35057
I20260928 15:01:52.951639 22089 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:52.951833 22089 heartbeater.cc:511] Master 127.1.252.254:35057 requested a full tablet report, sending...
I20260928 15:01:52.952256 21768 ts_manager.cc:194] Registered new tserver with Master: 4d2e0136566a4958887a5c7ca32341e1 (127.1.252.194:39717)
I20260928 15:01:52.952641 21768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.194:53763
I20260928 15:01:52.955523 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 21960
I20260928 15:01:52.955613 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-1/wal/instance
I20260928 15:01:52.955757 2035 external_mini_cluster.cc:1428] Setting key 24c80a86781e3356cdaa51875ecddc35
I20260928 15:01:52.956621 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.252.195:0
--local_ip_for_outbound_sockets=127.1.252.195
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35057
--builtin_ntp_servers=127.1.252.212:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:01:53.036188 22092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:53.036419 22092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:53.036455 22092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:53.037600 22092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:53.037662 22092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:53.037701 22092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.252.195
I20260928 15:01:53.039017 22092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.252.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.252.195
--webserver_port=0
--tserver_master_addrs=127.1.252.254:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.252.195
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:53.039283 22092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:53.039628 22092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:53.044672 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:53.044644 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:53.044823 22097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:53.044898 22092 server_base.cc:1034] running on GCE node
I20260928 15:01:53.045131 22092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:53.045320 22092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:53.046487 22092 hybrid_clock.cc:648] HybridClock initialized: now 1790607713046481 us; error 32 us; skew 500 ppm
I20260928 15:01:53.047710 22092 webserver.cc:460] Webserver started at http://127.1.252.195:38505/ using document root <none> and password file <none>
I20260928 15:01:53.047997 22092 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:53.048049 22092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:53.048192 22092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:53.049268 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data/instance:
uuid: "455330bd40a34860a3922d6ff61294c1"
format_stamp: "Formatted at 2026-09-28 15:01:53 on dist-test-slave-sf70"
server_key: "3c77a5714cbb1029632b5eb239ed7a0e"
server_key_iv: "e0894478819cc404eb60d4ddcfd451c4"
server_key_version: "encryptionkey@0"
I20260928 15:01:53.049628 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance:
uuid: "455330bd40a34860a3922d6ff61294c1"
format_stamp: "Formatted at 2026-09-28 15:01:53 on dist-test-slave-sf70"
server_key: "3c77a5714cbb1029632b5eb239ed7a0e"
server_key_iv: "e0894478819cc404eb60d4ddcfd451c4"
server_key_version: "encryptionkey@0"
I20260928 15:01:53.051023 22092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:01:53.051797 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:53.052042 22092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:01:53.052134 22092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal
uuid: "455330bd40a34860a3922d6ff61294c1"
format_stamp: "Formatted at 2026-09-28 15:01:53 on dist-test-slave-sf70"
server_key: "3c77a5714cbb1029632b5eb239ed7a0e"
server_key_iv: "e0894478819cc404eb60d4ddcfd451c4"
server_key_version: "encryptionkey@0"
I20260928 15:01:53.052223 22092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:53.063758 22092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:53.064092 22092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:53.064248 22092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:53.064458 22092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:01:53.064793 22092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:01:53.064855 22092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:53.064895 22092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:01:53.064921 22092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:53.070748 22092 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.195:35395
I20260928 15:01:53.070791 22220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.195:35395 every 8 connection(s)
I20260928 15:01:53.071159 22092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/data/info.pb
I20260928 15:01:53.076578 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.071874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.195:45233 (local address 127.1.252.254:35057)
0928 15:01:53.071963 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.071966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.072228 (+ 262us) server_negotiation.cc:407] Connection header received
0928 15:01:53.072697 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.072698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.072708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.072766 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:53.073407 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.073895 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.075114 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.075339 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.075407 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.076343 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:53.076345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:53.076345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:53.076350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:53.076365 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:53.076368 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:53.076412 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.076481 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.076505 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:01:53.076897 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.071847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35057 (local address 127.1.252.195:45233)
0928 15:01:53.072079 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.072141 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:53.072614 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:53.072815 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:53.072818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:53.072826 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:01:53.073338 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.073342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.073973 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.073974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.075025 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.075029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.075442 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.075443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.075569 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.076194 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:01:53.076195 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:01:53.076231 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:01:53.076254 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:01:53.076416 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:01:53.076419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:01:53.076421 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:01:53.076471 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:01:53.076551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":131,"threads_started":1}
I20260928 15:01:53.077479 22221 heartbeater.cc:347] Connected to a master server at 127.1.252.254:35057
I20260928 15:01:53.077607 22221 heartbeater.cc:464] Registering TS with master...
I20260928 15:01:53.077803 22221 heartbeater.cc:511] Master 127.1.252.254:35057 requested a full tablet report, sending...
I20260928 15:01:53.078193 21768 ts_manager.cc:194] Registered new tserver with Master: 455330bd40a34860a3922d6ff61294c1 (127.1.252.195:35395)
I20260928 15:01:53.078517 21768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.252.195:45233
I20260928 15:01:53.081238 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22092
I20260928 15:01:53.081323 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal/instance
I20260928 15:01:53.081465 2035 external_mini_cluster.cc:1428] Setting key 165d8f5b66913a034901749813c75024
I20260928 15:01:53.082280 2035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:01:53.086242 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.083610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.1.252.254:35057)
0928 15:01:53.083654 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.083655 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.083779 (+ 124us) server_negotiation.cc:407] Connection header received
0928 15:01:53.083831 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.083831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.083841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.083935 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:53.084086 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.084522 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.085259 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.085431 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.085479 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.086058 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:53.086059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:53.086059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:53.086063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:53.086075 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:53.086079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:53.086121 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.086172 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.086191 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 15:01:53.087716 21768 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:01:53.089366 21768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:01:53.095845 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.091701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.1.252.193:34109)
0928 15:01:53.091987 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.091992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.092002 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:01:53.092067 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.092069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.092087 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.092170 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:53.092389 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.093537 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.094796 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.095488 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.095551 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.095622 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.095720 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.095761 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":82,"threads_started":1}
I20260928 15:01:53.095856 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.091520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:39717 (local address 127.0.0.1:48630)
0928 15:01:53.092056 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.092064 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:53.092115 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:53.092423 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:53.092423 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:53.092425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:53.092466 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.092467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.093705 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.093706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.094621 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.094624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.095506 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.095507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.095605 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.095621 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:01:53.095697 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:01:53.095762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":109,"threads_started":1}
I20260928 15:01:53.096120 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.091768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.1.252.194:39717)
0928 15:01:53.091969 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.091974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.092170 (+ 196us) server_negotiation.cc:407] Connection header received
0928 15:01:53.092223 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.092225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.092239 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.092318 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:53.092537 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.093613 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.094700 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.095416 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.095508 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.095604 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.096000 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.096040 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260928 15:01:53.095856 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.091510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:34109 (local address 127.0.0.1:35932)
0928 15:01:53.091877 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.091908 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:53.091962 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:53.092209 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:53.092213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:53.092216 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:53.092285 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.092287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.093625 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.093626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.094723 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.094725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.095578 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.095578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.095626 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.095629 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:53.095697 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:01:53.095762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":27,"thread_start_us":75,"threads_started":1}
I20260928 15:01:53.096779 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 949add4a01bc4ee096029abd96cd77b9 (DEFAULT_TABLE table=test-table [id=946bcd18dc2249129b71c120edb6692d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:53.096869 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.091510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35395 (local address 127.0.0.1:36320)
0928 15:01:53.091787 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.091827 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:53.091949 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:53.092484 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:53.092485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:53.092487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:53.092520 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.092521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.093353 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.093354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.094721 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.094724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.096605 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.096605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.096676 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.096679 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:53.096758 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:01:53.096797 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":24,"thread_start_us":119,"threads_started":1}
I20260928 15:01:53.096891 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.091678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.1.252.195:35395)
0928 15:01:53.092288 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.092292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.092302 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:01:53.092337 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.092339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.092354 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.092425 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:53.092590 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.093264 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.095664 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.096426 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.096487 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.096571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.096777 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.096812 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":75,"threads_started":1}
I20260928 15:01:53.097149 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949add4a01bc4ee096029abd96cd77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:53.097503 22154 tablet_service.cc:1467] Processing CreateTablet for tablet 949add4a01bc4ee096029abd96cd77b9 (DEFAULT_TABLE table=test-table [id=946bcd18dc2249129b71c120edb6692d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:53.097815 22154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949add4a01bc4ee096029abd96cd77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:53.099874 22235 tablet_bootstrap.cc:492] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: Bootstrap starting.
I20260928 15:01:53.100353 22236 tablet_bootstrap.cc:492] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Bootstrap starting.
I20260928 15:01:53.100608 22235 tablet_bootstrap.cc:654] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:53.100728 21890 tablet_service.cc:1467] Processing CreateTablet for tablet 949add4a01bc4ee096029abd96cd77b9 (DEFAULT_TABLE table=test-table [id=946bcd18dc2249129b71c120edb6692d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:01:53.100935 22235 log.cc:824] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:53.100971 22236 tablet_bootstrap.cc:654] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:53.101068 21890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949add4a01bc4ee096029abd96cd77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:53.101281 22236 log.cc:824] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:53.101665 22235 tablet_bootstrap.cc:492] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: No bootstrap required, opened a new log
I20260928 15:01:53.101735 22235 ts_tablet_manager.cc:1397] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:53.101949 22236 tablet_bootstrap.cc:492] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: No bootstrap required, opened a new log
I20260928 15:01:53.102034 22236 ts_tablet_manager.cc:1397] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:53.103751 22236 raft_consensus.cc:348] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.103921 22236 raft_consensus.cc:374] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:53.103977 22236 raft_consensus.cc:729] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455330bd40a34860a3922d6ff61294c1, State: Initialized, Role: FOLLOWER
I20260928 15:01:53.104094 22236 consensus_queue.cc:260] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.104364 22236 ts_tablet_manager.cc:1428] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:01:53.104486 22221 heartbeater.cc:503] Master 127.1.252.254:35057 was elected leader, sending a full tablet report...
I20260928 15:01:53.104417 22235 raft_consensus.cc:348] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.104578 22235 raft_consensus.cc:374] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:53.104481 22240 tablet_bootstrap.cc:492] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: Bootstrap starting.
I20260928 15:01:53.104629 22235 raft_consensus.cc:729] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2e0136566a4958887a5c7ca32341e1, State: Initialized, Role: FOLLOWER
I20260928 15:01:53.104773 22235 consensus_queue.cc:260] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.105036 22235 ts_tablet_manager.cc:1428] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 15:01:53.105147 22089 heartbeater.cc:503] Master 127.1.252.254:35057 was elected leader, sending a full tablet report...
I20260928 15:01:53.105306 22240 tablet_bootstrap.cc:654] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:53.105631 22240 log.cc:824] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:53.106416 22240 tablet_bootstrap.cc:492] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: No bootstrap required, opened a new log
I20260928 15:01:53.106515 22240 ts_tablet_manager.cc:1397] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:01:53.108389 22240 raft_consensus.cc:348] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.108516 22240 raft_consensus.cc:374] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:53.108538 22240 raft_consensus.cc:729] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500798626d92407caf58851b62b4724b, State: Initialized, Role: FOLLOWER
I20260928 15:01:53.108639 22240 consensus_queue.cc:260] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.108923 22240 ts_tablet_manager.cc:1428] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:01:53.108983 21957 heartbeater.cc:503] Master 127.1.252.254:35057 was elected leader, sending a full tablet report...
W20260928 15:01:53.196259 22090 tablet.cc:2369] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:53.321696 22222 tablet.cc:2369] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:01:53.324474 21958 tablet.cc:2369] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:01:53.396970 22241 raft_consensus.cc:482] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:01:53.397101 22241 raft_consensus.cc:504] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.397585 22241 leader_election.cc:288] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 500798626d92407caf58851b62b4724b (127.1.252.193:34109), 455330bd40a34860a3922d6ff61294c1 (127.1.252.195:35395)
I20260928 15:01:53.400701 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.397953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:38807 (local address 127.1.252.193:34109)
0928 15:01:53.398015 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.398017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.398124 (+ 107us) server_negotiation.cc:407] Connection header received
0928 15:01:53.398205 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.398206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.398224 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.398290 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:53.398453 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.398917 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.399993 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.400344 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.400428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.400437 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.400586 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.400619 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 15:01:53.400732 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.397968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.252.194:49959 (local address 127.1.252.195:35395)
0928 15:01:53.398015 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.398017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.398024 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:01:53.398067 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.398067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.398083 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.398144 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:01:53.398355 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.398937 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.399998 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.400362 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.400459 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.400468 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.400618 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.400664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:01:53.400732 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.397788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.195:35395 (local address 127.1.252.194:49959)
0928 15:01:53.397880 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.397895 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:53.397963 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:53.398162 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:53.398163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:53.398165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:53.398240 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.398242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.399025 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.399026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.399834 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.399836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.400458 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.400459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.400534 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.400537 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:01:53.400604 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 15:01:53.400646 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:01:53.400741 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.397788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:34109 (local address 127.1.252.194:38807)
0928 15:01:53.398044 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.398060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:01:53.398138 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:01:53.398293 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:01:53.398294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:01:53.398295 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:01:53.398374 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.398375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.399037 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.399038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.399872 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:01:53.399875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.400428 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.400429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:01:53.400506 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.400510 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:01:53.400579 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:01:53.400619 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":53,"thread_start_us":112,"threads_started":1}
I20260928 15:01:53.401136 22174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949add4a01bc4ee096029abd96cd77b9" candidate_uuid: "4d2e0136566a4958887a5c7ca32341e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "455330bd40a34860a3922d6ff61294c1" is_pre_election: true
I20260928 15:01:53.401131 21910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949add4a01bc4ee096029abd96cd77b9" candidate_uuid: "4d2e0136566a4958887a5c7ca32341e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500798626d92407caf58851b62b4724b" is_pre_election: true
I20260928 15:01:53.401292 21910 raft_consensus.cc:2454] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d2e0136566a4958887a5c7ca32341e1 in term 0.
I20260928 15:01:53.401292 22174 raft_consensus.cc:2454] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d2e0136566a4958887a5c7ca32341e1 in term 0.
I20260928 15:01:53.401602 21978 leader_election.cc:302] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d2e0136566a4958887a5c7ca32341e1, 500798626d92407caf58851b62b4724b; no voters:
I20260928 15:01:53.401815 22241 raft_consensus.cc:2781] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:01:53.401912 22241 raft_consensus.cc:482] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:01:53.401952 22241 raft_consensus.cc:3037] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:53.402722 22241 raft_consensus.cc:504] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.402892 22241 leader_election.cc:288] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [CANDIDATE]: Term 1 election: Requested vote from peers 500798626d92407caf58851b62b4724b (127.1.252.193:34109), 455330bd40a34860a3922d6ff61294c1 (127.1.252.195:35395)
I20260928 15:01:53.403102 22174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949add4a01bc4ee096029abd96cd77b9" candidate_uuid: "4d2e0136566a4958887a5c7ca32341e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "455330bd40a34860a3922d6ff61294c1"
I20260928 15:01:53.403198 22174 raft_consensus.cc:3037] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:53.403966 22174 raft_consensus.cc:2454] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d2e0136566a4958887a5c7ca32341e1 in term 1.
I20260928 15:01:53.403990 21910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949add4a01bc4ee096029abd96cd77b9" candidate_uuid: "4d2e0136566a4958887a5c7ca32341e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500798626d92407caf58851b62b4724b"
I20260928 15:01:53.404088 21910 raft_consensus.cc:3037] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:53.404230 21977 leader_election.cc:302] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 455330bd40a34860a3922d6ff61294c1, 4d2e0136566a4958887a5c7ca32341e1; no voters:
I20260928 15:01:53.404336 22241 raft_consensus.cc:2781] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:53.404515 22241 raft_consensus.cc:686] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 1 LEADER]: Becoming Leader. State: Replica: 4d2e0136566a4958887a5c7ca32341e1, State: Running, Role: LEADER
I20260928 15:01:53.404616 22241 consensus_queue.cc:237] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } }
I20260928 15:01:53.404999 21910 raft_consensus.cc:2454] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d2e0136566a4958887a5c7ca32341e1 in term 1.
I20260928 15:01:53.405491 21767 catalog_manager.cc:5495] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d2e0136566a4958887a5c7ca32341e1 (127.1.252.194). New cstate: current_term: 1 leader_uuid: "4d2e0136566a4958887a5c7ca32341e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d2e0136566a4958887a5c7ca32341e1" member_type: VOTER last_known_addr { host: "127.1.252.194" port: 39717 } health_report { overall_health: HEALTHY } } }
I20260928 15:01:53.457216 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.454809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.1.252.194:39717)
0928 15:01:53.454874 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:53.454876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:53.454885 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:01:53.454930 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:53.454930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:53.454947 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:53.455008 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:01:53.455137 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.455592 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.456371 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:53.456585 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:53.456649 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:53.456811 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:01:53.456941 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:01:53.457005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:53.457076 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 15:01:53.457098 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:01:53.459199 21910 raft_consensus.cc:1261] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 1 FOLLOWER]: Refusing update from remote peer 4d2e0136566a4958887a5c7ca32341e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:53.459236 22174 raft_consensus.cc:1261] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 1 FOLLOWER]: Refusing update from remote peer 4d2e0136566a4958887a5c7ca32341e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:01:53.459584 22245 consensus_queue.cc:1035] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500798626d92407caf58851b62b4724b" member_type: VOTER last_known_addr { host: "127.1.252.193" port: 34109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:53.459738 22241 consensus_queue.cc:1035] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455330bd40a34860a3922d6ff61294c1" member_type: VOTER last_known_addr { host: "127.1.252.195" port: 35395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:01:53.462267 22250 mvcc.cc:205] Tried to move back new op lower bound from 7334329194325901312 to 7334329194105774080. Current Snapshot: MvccSnapshot[applied={T|T < 7334329194325901312}]
I20260928 15:01:53.464041 21768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:53.464140 21768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:01:53.464887 21768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88606b9879624c6b9e1c8720e15741cc: Sending DeleteTablet for 3 replicas of tablet 949add4a01bc4ee096029abd96cd77b9
I20260928 15:01:53.465189 21890 tablet_service.cc:1514] Processing DeleteTablet for tablet 949add4a01bc4ee096029abd96cd77b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:53 UTC) from {username='slave'} at 127.0.0.1:35932
I20260928 15:01:53.465232 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 949add4a01bc4ee096029abd96cd77b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:53 UTC) from {username='slave'} at 127.0.0.1:48630
I20260928 15:01:53.465332 22154 tablet_service.cc:1514] Processing DeleteTablet for tablet 949add4a01bc4ee096029abd96cd77b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:01:53 UTC) from {username='slave'} at 127.0.0.1:36320
I20260928 15:01:53.465466 22258 tablet_replica.cc:330] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1: stopping tablet replica
I20260928 15:01:53.465566 22258 raft_consensus.cc:2229] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:01:53.465600 22257 tablet_replica.cc:330] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b: stopping tablet replica
I20260928 15:01:53.465660 22258 raft_consensus.cc:2258] T 949add4a01bc4ee096029abd96cd77b9 P 4d2e0136566a4958887a5c7ca32341e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:53.465690 22257 raft_consensus.cc:2229] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:53.465778 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21828
I20260928 15:01:53.465797 22257 raft_consensus.cc:2258] T 949add4a01bc4ee096029abd96cd77b9 P 500798626d92407caf58851b62b4724b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 15:01:53.471953 21755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.252.193:34109 (error 108)
I20260928 15:01:53.472241 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21960
I20260928 15:01:53.472985 22259 tablet_replica.cc:330] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: stopping tablet replica
I20260928 15:01:53.473117 22259 raft_consensus.cc:2229] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:01:53.473285 22259 raft_consensus.cc:2258] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:01:53.473845 22259 ts_tablet_manager.cc:1905] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:01:53.474539 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.474353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.193:34109 (local address 127.0.0.1:35942)
0928 15:01:53.474432 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.474505 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.193:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 15:01:53.474861 21755 catalog_manager.cc:4584] TS 500798626d92407caf58851b62b4724b (127.1.252.193:34109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 949add4a01bc4ee096029abd96cd77b9: Network error: Client connection negotiation failed: client connection to 127.1.252.193:34109: connect: Connection refused (error 111)
I20260928 15:01:53.475837 22259 ts_tablet_manager.cc:1918] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:01:53.475937 22259 log.cc:1196] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Deleting WAL directory at /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607474232031-2035-0/minicluster-data/ts-2/wal/wals/949add4a01bc4ee096029abd96cd77b9
I20260928 15:01:53.476209 22259 ts_tablet_manager.cc:1939] T 949add4a01bc4ee096029abd96cd77b9 P 455330bd40a34860a3922d6ff61294c1: Deleting consensus metadata
I20260928 15:01:53.476639 21754 catalog_manager.cc:4841] TS 455330bd40a34860a3922d6ff61294c1 (127.1.252.195:35395): tablet 949add4a01bc4ee096029abd96cd77b9 (table test-table [id=946bcd18dc2249129b71c120edb6692d]) successfully deleted
I20260928 15:01:53.479656 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:53.479520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.194:39717 (local address 127.0.0.1:48648)
0928 15:01:53.479583 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 15:01:53.479616 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.194:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 15:01:53.479928 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22092
I20260928 15:01:53.484668 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 21737
2026-09-28T15:01:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (960 ms)
[----------] 6 tests from EncryptionPolicyTest (8733 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:53 dist-test-slave-sf70 krb5kdc[22263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:53 dist-test-slave-sf70 krb5kdc[22263](info): set up 2 sockets
Sep 28 15:01:53 dist-test-slave-sf70 krb5kdc[22263](info): commencing operation
krb5kdc: starting...
W20260928 15:01:54.557660 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:54 dist-test-slave-sf70 krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:54.679034 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:54.759186 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:54.759420 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:54.759459 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:54.760690 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:54.760757 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:54.760782 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:54.760809 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:54.760839 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:54.762398 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:54.762681 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:54.762908 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:54.767788 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:54.767782 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:54.767782 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:54.767941 22276 server_base.cc:1034] running on GCE node
I20260928 15:01:54.768277 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:54.768633 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:54.769791 22276 hybrid_clock.cc:648] HybridClock initialized: now 1790607714769772 us; error 40 us; skew 500 ppm
Sep 28 15:01:54 dist-test-slave-sf70 krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:54.772217 22276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:54.772714 22276 webserver.cc:460] Webserver started at http://127.0.0.1:44971/ using document root <none> and password file <none>
I20260928 15:01:54.773007 22276 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:54.773085 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:54.773248 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:54.774353 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "c57c997b51a14bbca759180f50fd000f"
format_stamp: "Formatted at 2026-09-28 15:01:54 on dist-test-slave-sf70"
server_key: "317aa9944accaeecd7f24cda58ca57b5"
server_key_iv: "11c2a4f07a7224e1967514134403788e"
server_key_version: "encryptionkey@0"
I20260928 15:01:54.774742 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "c57c997b51a14bbca759180f50fd000f"
format_stamp: "Formatted at 2026-09-28 15:01:54 on dist-test-slave-sf70"
server_key: "317aa9944accaeecd7f24cda58ca57b5"
server_key_iv: "11c2a4f07a7224e1967514134403788e"
server_key_version: "encryptionkey@0"
I20260928 15:01:54.776311 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:01:54.777104 22291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:54.777273 22276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 15:01:54.777374 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "c57c997b51a14bbca759180f50fd000f"
format_stamp: "Formatted at 2026-09-28 15:01:54 on dist-test-slave-sf70"
server_key: "317aa9944accaeecd7f24cda58ca57b5"
server_key_iv: "11c2a4f07a7224e1967514134403788e"
server_key_version: "encryptionkey@0"
I20260928 15:01:54.777500 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:54.790661 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:54.792546 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:54.792706 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:54.796779 22276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45091
I20260928 15:01:54.796808 22344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45091 every 8 connection(s)
I20260928 15:01:54.797150 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:54.798125 22345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:54.800472 22345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f: Bootstrap starting.
I20260928 15:01:54.801065 22345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:54.801360 22345 log.cc:824] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:54.802174 22345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f: No bootstrap required, opened a new log
I20260928 15:01:54.803512 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22276
I20260928 15:01:54.803614 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:54.803993 2035 external_mini_cluster.cc:1428] Setting key 1b5083be60e684c6fdd866f072e07d9f
I20260928 15:01:54.803833 22345 raft_consensus.cc:348] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57c997b51a14bbca759180f50fd000f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260928 15:01:54.804059 22345 raft_consensus.cc:374] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:54.804108 22345 raft_consensus.cc:729] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c997b51a14bbca759180f50fd000f, State: Initialized, Role: FOLLOWER
I20260928 15:01:54.804275 22345 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57c997b51a14bbca759180f50fd000f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260928 15:01:54.804366 22345 raft_consensus.cc:388] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:54.804404 22345 raft_consensus.cc:482] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:54.804471 22345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:54.805488 22345 raft_consensus.cc:504] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57c997b51a14bbca759180f50fd000f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260928 15:01:54.805574 22345 leader_election.cc:302] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57c997b51a14bbca759180f50fd000f; no voters:
I20260928 15:01:54.805738 22345 leader_election.cc:288] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:54.805794 22350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:54.805958 22350 raft_consensus.cc:686] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [term 1 LEADER]: Becoming Leader. State: Replica: c57c997b51a14bbca759180f50fd000f, State: Running, Role: LEADER
I20260928 15:01:54.806015 22345 sys_catalog.cc:564] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:54.806052 22350 consensus_queue.cc:237] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57c997b51a14bbca759180f50fd000f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260928 15:01:54.807509 22352 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c57c997b51a14bbca759180f50fd000f. Latest consensus state: current_term: 1 leader_uuid: "c57c997b51a14bbca759180f50fd000f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57c997b51a14bbca759180f50fd000f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } } }
I20260928 15:01:54.807605 22352 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:54.807705 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c57c997b51a14bbca759180f50fd000f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57c997b51a14bbca759180f50fd000f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } } }
I20260928 15:01:54.807798 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f [sys.catalog]: This master's current role is: LEADER
W20260928 15:01:54.808079 22364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:01:54.808137 22364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:01:54.808207 22365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:54.808385 22365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:54.809916 22365 catalog_manager.cc:1347] Generated new cluster ID: c3229319908d4f9bbb055c47fd6c50c2
I20260928 15:01:54.809978 22365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:01:54 dist-test-slave-sf70 krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:01:54.815672 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:54.804595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55459 (local address 127.0.0.1:45091)
0928 15:01:54.804809 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:54.804819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:54.804853 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:01:54.804983 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:54.804992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:54.805057 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:54.805225 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:54.806017 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.807554 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.808702 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.809112 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.809229 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:54.811877 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:54.811880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:54.811881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:54.811933 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:54.813833 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:54.814231 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:54.814234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:54.814234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:54.814337 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:54.814559 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:54.814560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:54.814560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:54.814823 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:54.814966 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:54.815381 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 15:01:54.815487 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260928 15:01:54.816807 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:54.825618 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:54.818381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44529 (local address 127.0.0.1:45091)
0928 15:01:54.818418 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:54.818420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:54.818520 (+ 100us) server_negotiation.cc:407] Connection header received
0928 15:01:54.818582 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:54.818582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:54.818591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:54.818640 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:54.819301 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.820028 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.821020 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:54.821320 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:54.821377 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:54.822837 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:54.822838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:54.822838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:54.822844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:54.824350 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:54.824788 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:54.824789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:54.824789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:54.824843 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:54.825085 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:54.825085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:54.825085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:54.825198 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:54.825295 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:54.825469 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 15:01:54.825531 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:01:54.841866 22365 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:54.842394 22365 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:54.851318 22365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c57c997b51a14bbca759180f50fd000f: Generated new TSK 0
I20260928 15:01:54.851531 22365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:54.860692 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22276
W20260928 15:01:54.865046 22261 connection.cc:535] client connection to 127.0.0.1:45091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45091): Connection reset by peer (error 104)
2026-09-28T15:01:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:54 dist-test-slave-sf70 krb5kdc[22376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:54 dist-test-slave-sf70 krb5kdc[22376](info): set up 2 sockets
Sep 28 15:01:54 dist-test-slave-sf70 krb5kdc[22376](info): commencing operation
krb5kdc: starting...
W20260928 15:01:55.925523 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:56 dist-test-slave-sf70 krb5kdc[22376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:56.046042 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:56.126577 22389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:56.126816 22389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:56.126861 22389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:56.128065 22389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:56.128137 22389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:56.128161 22389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:56.128182 22389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:56.128212 22389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:56.129647 22389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:56.129920 22389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:56.130141 22389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:56.134872 22394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:56.134883 22397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:56.134903 22395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:56.135272 22389 server_base.cc:1034] running on GCE node
I20260928 15:01:56.135488 22389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:56.135735 22389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:56.136870 22389 hybrid_clock.cc:648] HybridClock initialized: now 1790607716136849 us; error 40 us; skew 500 ppm
Sep 28 15:01:56 dist-test-slave-sf70 krb5kdc[22376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:56.139313 22389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:56.139741 22389 webserver.cc:460] Webserver started at http://127.0.0.1:41383/ using document root <none> and password file <none>
I20260928 15:01:56.140057 22389 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:56.140127 22389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:56.140283 22389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:56.141314 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "dc702f115e7747fdab405f169b666b2a"
format_stamp: "Formatted at 2026-09-28 15:01:56 on dist-test-slave-sf70"
server_key: "4ec1550d7fcabad6d30036a711697ab5"
server_key_iv: "3503c1f1275f62010614d505096e29d7"
server_key_version: "encryptionkey@0"
I20260928 15:01:56.141676 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "dc702f115e7747fdab405f169b666b2a"
format_stamp: "Formatted at 2026-09-28 15:01:56 on dist-test-slave-sf70"
server_key: "4ec1550d7fcabad6d30036a711697ab5"
server_key_iv: "3503c1f1275f62010614d505096e29d7"
server_key_version: "encryptionkey@0"
I20260928 15:01:56.143009 22389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:56.143714 22404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:56.143868 22389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:01:56.144001 22389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "dc702f115e7747fdab405f169b666b2a"
format_stamp: "Formatted at 2026-09-28 15:01:56 on dist-test-slave-sf70"
server_key: "4ec1550d7fcabad6d30036a711697ab5"
server_key_iv: "3503c1f1275f62010614d505096e29d7"
server_key_version: "encryptionkey@0"
I20260928 15:01:56.144089 22389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:56.169438 22389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:56.171711 22389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:56.171907 22389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:56.176031 22389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36157
I20260928 15:01:56.176110 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36157 every 8 connection(s)
I20260928 15:01:56.176474 22389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:56.177506 22458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:56.179850 22458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a: Bootstrap starting.
I20260928 15:01:56.180480 22458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:56.180768 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22389
I20260928 15:01:56.180830 22458 log.cc:824] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:56.180886 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:56.181082 2035 external_mini_cluster.cc:1428] Setting key 64eb7f2755e090fcf92a1c8d3b43509f
I20260928 15:01:56.181733 22458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a: No bootstrap required, opened a new log
I20260928 15:01:56.184442 22458 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc702f115e7747fdab405f169b666b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36157 } }
I20260928 15:01:56.184628 22458 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:56.184660 22458 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc702f115e7747fdab405f169b666b2a, State: Initialized, Role: FOLLOWER
I20260928 15:01:56.184791 22458 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc702f115e7747fdab405f169b666b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36157 } }
I20260928 15:01:56.184886 22458 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:56.185019 22458 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:56.185096 22458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:56.185910 22458 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc702f115e7747fdab405f169b666b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36157 } }
I20260928 15:01:56.186050 22458 leader_election.cc:302] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc702f115e7747fdab405f169b666b2a; no voters:
I20260928 15:01:56.186228 22458 leader_election.cc:288] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:56.186313 22463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:56.186462 22463 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [term 1 LEADER]: Becoming Leader. State: Replica: dc702f115e7747fdab405f169b666b2a, State: Running, Role: LEADER
I20260928 15:01:56.186636 22463 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc702f115e7747fdab405f169b666b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36157 } }
I20260928 15:01:56.186501 22458 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:56.187026 22465 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc702f115e7747fdab405f169b666b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc702f115e7747fdab405f169b666b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36157 } } }
I20260928 15:01:56.187067 22464 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc702f115e7747fdab405f169b666b2a. Latest consensus state: current_term: 1 leader_uuid: "dc702f115e7747fdab405f169b666b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc702f115e7747fdab405f169b666b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36157 } } }
I20260928 15:01:56.187140 22465 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:56.187253 22464 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:56.187417 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:56.181613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55093 (local address 127.0.0.1:36157)
0928 15:01:56.181794 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:56.181804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:56.181842 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:01:56.182303 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:56.182309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:56.182357 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:56.182475 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:56.182717 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:56.183610 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:56.184836 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:56.185408 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:56.185522 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:56.186402 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:56.186406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:56.186407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:56.186628 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:56.186668 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:56.186690 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:56.186791 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:56.186913 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 15:01:56.186947 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260928 15:01:56.187459 22467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:56.188563 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:56.188596 22467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:56.193328 22467 catalog_manager.cc:1347] Generated new cluster ID: 03716d00343441db905dbf540a7d2b8b
I20260928 15:01:56.193336 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:56.189963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33069 (local address 127.0.0.1:36157)
0928 15:01:56.189999 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:56.190000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:56.190190 (+ 190us) server_negotiation.cc:407] Connection header received
0928 15:01:56.190252 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:56.190253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:56.190262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:56.190308 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:56.190468 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:56.190944 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:56.191988 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:56.192268 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:56.192342 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:56.193073 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:56.193074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:56.193074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:56.193078 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:56.193092 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:56.193096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:56.193152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:56.193230 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:01:56.193240 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:01:56.193391 22467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:56.220620 22467 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:56.221199 22467 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:56.223228 22467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc702f115e7747fdab405f169b666b2a: Generated new TSK 0
I20260928 15:01:56.223410 22467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:56.229238 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22389
W20260928 15:01:56.233651 22374 connection.cc:535] client connection to 127.0.0.1:36157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36157): Connection reset by peer (error 104)
2026-09-28T15:01:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:56 dist-test-slave-sf70 krb5kdc[22489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:56 dist-test-slave-sf70 krb5kdc[22489](info): set up 2 sockets
Sep 28 15:01:56 dist-test-slave-sf70 krb5kdc[22489](info): commencing operation
krb5kdc: starting...
W20260928 15:01:57.282294 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:57 dist-test-slave-sf70 krb5kdc[22489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:57.407264 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:57.491736 22502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:57.492044 22502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:57.492089 22502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:57.493251 22502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:57.493319 22502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:57.493343 22502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:57.493381 22502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:57.493412 22502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:57.494897 22502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:57.495182 22502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:57.495415 22502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:01:57.500268 22510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:57.500272 22507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:57.500365 22508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:57.500497 22502 server_base.cc:1034] running on GCE node
I20260928 15:01:57.500736 22502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:57.501032 22502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:57.502205 22502 hybrid_clock.cc:648] HybridClock initialized: now 1790607717502203 us; error 42 us; skew 500 ppm
Sep 28 15:01:57 dist-test-slave-sf70 krb5kdc[22489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:57.504812 22502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:57.505296 22502 webserver.cc:460] Webserver started at http://127.0.0.1:34009/ using document root <none> and password file <none>
I20260928 15:01:57.505486 22502 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:57.505519 22502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:57.505612 22502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:57.506603 22502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "aca6f9316aba46bd916e2bf2404346c3"
format_stamp: "Formatted at 2026-09-28 15:01:57 on dist-test-slave-sf70"
server_key: "137803005ac8dcc4ed5613525285ad5c"
server_key_iv: "c8e7ad80ccc83a2034cfab2cc1d696b1"
server_key_version: "encryptionkey@0"
I20260928 15:01:57.506935 22502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "aca6f9316aba46bd916e2bf2404346c3"
format_stamp: "Formatted at 2026-09-28 15:01:57 on dist-test-slave-sf70"
server_key: "137803005ac8dcc4ed5613525285ad5c"
server_key_iv: "c8e7ad80ccc83a2034cfab2cc1d696b1"
server_key_version: "encryptionkey@0"
I20260928 15:01:57.508245 22502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:01:57.509051 22517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:57.509277 22502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:57.509362 22502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "aca6f9316aba46bd916e2bf2404346c3"
format_stamp: "Formatted at 2026-09-28 15:01:57 on dist-test-slave-sf70"
server_key: "137803005ac8dcc4ed5613525285ad5c"
server_key_iv: "c8e7ad80ccc83a2034cfab2cc1d696b1"
server_key_version: "encryptionkey@0"
I20260928 15:01:57.509425 22502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:57.513098 22502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:57.515309 22502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:57.515425 22502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:57.519232 22502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33935
I20260928 15:01:57.519316 22570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33935 every 8 connection(s)
I20260928 15:01:57.519658 22502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:57.520717 22571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:57.522055 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22502
I20260928 15:01:57.522198 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:57.522387 2035 external_mini_cluster.cc:1428] Setting key 3952292a70e2f6eec77c397878af8776
I20260928 15:01:57.523536 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3: Bootstrap starting.
I20260928 15:01:57.524282 22571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:57.524638 22571 log.cc:824] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:57.525017 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:57.522913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33619 (local address 127.0.0.1:33935)
0928 15:01:57.523112 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:57.523121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:57.523158 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:01:57.523686 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:57.523694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:57.523725 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:57.523945 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:57.524530 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:57.524533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:57.524534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:57.524538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:57.524582 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:57.524597 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:57.524689 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:57.524720 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 15:01:57.524866 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 15:01:57.525334 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3: No bootstrap required, opened a new log
I20260928 15:01:57.525995 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:57.527403 22571 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca6f9316aba46bd916e2bf2404346c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33935 } }
I20260928 15:01:57.527554 22571 raft_consensus.cc:374] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:57.527598 22571 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca6f9316aba46bd916e2bf2404346c3, State: Initialized, Role: FOLLOWER
I20260928 15:01:57.527721 22571 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca6f9316aba46bd916e2bf2404346c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33935 } }
I20260928 15:01:57.527814 22571 raft_consensus.cc:388] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:57.527858 22571 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:57.527933 22571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:57.528697 22571 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca6f9316aba46bd916e2bf2404346c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33935 } }
I20260928 15:01:57.528828 22571 leader_election.cc:302] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aca6f9316aba46bd916e2bf2404346c3; no voters:
I20260928 15:01:57.528995 22571 leader_election.cc:288] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:57.529043 22581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:57.529201 22581 raft_consensus.cc:686] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [term 1 LEADER]: Becoming Leader. State: Replica: aca6f9316aba46bd916e2bf2404346c3, State: Running, Role: LEADER
I20260928 15:01:57.529261 22571 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:57.529328 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:57.528164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57441 (local address 127.0.0.1:33935)
0928 15:01:57.528203 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:57.528205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:57.528324 (+ 119us) server_negotiation.cc:407] Connection header received
0928 15:01:57.528392 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:57.528393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:57.528398 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:57.528442 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:57.529001 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:57.529002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:57.529002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:01:57.529005 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:57.529017 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:01:57.529021 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:57.529095 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:57.529188 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 15:01:57.529201 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:01:57.529338 22581 consensus_queue.cc:237] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca6f9316aba46bd916e2bf2404346c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33935 } }
I20260928 15:01:57.530069 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aca6f9316aba46bd916e2bf2404346c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca6f9316aba46bd916e2bf2404346c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33935 } } }
I20260928 15:01:57.530077 22582 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca6f9316aba46bd916e2bf2404346c3. Latest consensus state: current_term: 1 leader_uuid: "aca6f9316aba46bd916e2bf2404346c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca6f9316aba46bd916e2bf2404346c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33935 } } }
I20260928 15:01:57.530205 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:57.530216 22582 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3 [sys.catalog]: This master's current role is: LEADER
W20260928 15:01:57.530983 22596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:01:57.531047 22596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:01:57.531104 22592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:57.531255 22592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:57.532725 22592 catalog_manager.cc:1347] Generated new cluster ID: 706f053171f7436aa7c042a42654fa89
I20260928 15:01:57.532781 22592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:01:57.566334 22592 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:57.566576 22592 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:57.572932 22592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aca6f9316aba46bd916e2bf2404346c3: Generated new TSK 0
I20260928 15:01:57.573146 22592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:57.579635 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22502
2026-09-28T15:01:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:57 dist-test-slave-sf70 krb5kdc[22602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:57 dist-test-slave-sf70 krb5kdc[22602](info): set up 2 sockets
Sep 28 15:01:57 dist-test-slave-sf70 krb5kdc[22602](info): commencing operation
krb5kdc: starting...
W20260928 15:01:58.633981 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:01:58 dist-test-slave-sf70 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:01:58.758667 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:01:58.844758 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:01:58.844961 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:01:58.844985 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:01:58.846035 22615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:01:58.846068 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:01:58.846082 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:01:58.846095 22615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:01:58.846107 22615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:01:58.847476 22615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:01:58.847675 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:01:58.847841 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:01:58.853010 22615 server_base.cc:1034] running on GCE node
W20260928 15:01:58.852949 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:58.852957 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:01:58.853012 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:01:58.853506 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:01:58.853823 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:01:58.854974 22615 hybrid_clock.cc:648] HybridClock initialized: now 1790607718854968 us; error 37 us; skew 500 ppm
Sep 28 15:01:58 dist-test-slave-sf70 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:01:58.857527 22615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:01:58.857985 22615 webserver.cc:460] Webserver started at http://127.0.0.1:42129/ using document root <none> and password file <none>
I20260928 15:01:58.858215 22615 fs_manager.cc:359] Metadata directory not provided
I20260928 15:01:58.858260 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:01:58.858413 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:01:58.859503 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "aa7d184d1c3345898c902a39ffbf80e3"
format_stamp: "Formatted at 2026-09-28 15:01:58 on dist-test-slave-sf70"
server_key: "b8dd1f1b8bb2178e99b0d36d454bb5ee"
server_key_iv: "e0fff9b0593fff02f5c8ad15424bbfd9"
server_key_version: "encryptionkey@0"
I20260928 15:01:58.859860 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "aa7d184d1c3345898c902a39ffbf80e3"
format_stamp: "Formatted at 2026-09-28 15:01:58 on dist-test-slave-sf70"
server_key: "b8dd1f1b8bb2178e99b0d36d454bb5ee"
server_key_iv: "e0fff9b0593fff02f5c8ad15424bbfd9"
server_key_version: "encryptionkey@0"
I20260928 15:01:58.861223 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:58.862047 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:01:58.862255 22615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:01:58.862371 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "aa7d184d1c3345898c902a39ffbf80e3"
format_stamp: "Formatted at 2026-09-28 15:01:58 on dist-test-slave-sf70"
server_key: "b8dd1f1b8bb2178e99b0d36d454bb5ee"
server_key_iv: "e0fff9b0593fff02f5c8ad15424bbfd9"
server_key_version: "encryptionkey@0"
I20260928 15:01:58.862495 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:01:58.896975 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:01:58.899214 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:01:58.899385 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:01:58.903772 22615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46009
I20260928 15:01:58.903837 22682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46009 every 8 connection(s)
I20260928 15:01:58.904233 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:01:58.905233 22683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:01:58.907447 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3: Bootstrap starting.
I20260928 15:01:58.908054 22683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3: Neither blocks nor log segments found. Creating new log.
I20260928 15:01:58.908293 22683 log.cc:824] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:01:58.908998 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3: No bootstrap required, opened a new log
I20260928 15:01:58.910624 22683 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7d184d1c3345898c902a39ffbf80e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 15:01:58.910761 22683 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:01:58.910784 22683 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7d184d1c3345898c902a39ffbf80e3, State: Initialized, Role: FOLLOWER
I20260928 15:01:58.910881 22683 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7d184d1c3345898c902a39ffbf80e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 15:01:58.910933 22683 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:01:58.910952 22683 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:01:58.910979 22683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:01:58.911470 22683 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7d184d1c3345898c902a39ffbf80e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 15:01:58.911562 22683 leader_election.cc:302] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa7d184d1c3345898c902a39ffbf80e3; no voters:
I20260928 15:01:58.911689 22683 leader_election.cc:288] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:01:58.911767 22686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:01:58.911993 22686 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [term 1 LEADER]: Becoming Leader. State: Replica: aa7d184d1c3345898c902a39ffbf80e3, State: Running, Role: LEADER
I20260928 15:01:58.912024 22683 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:01:58.912102 22686 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7d184d1c3345898c902a39ffbf80e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 15:01:58.912680 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa7d184d1c3345898c902a39ffbf80e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7d184d1c3345898c902a39ffbf80e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } } }
I20260928 15:01:58.912778 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:58.913054 22695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:01:58.913223 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa7d184d1c3345898c902a39ffbf80e3. Latest consensus state: current_term: 1 leader_uuid: "aa7d184d1c3345898c902a39ffbf80e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7d184d1c3345898c902a39ffbf80e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } } }
I20260928 15:01:58.913323 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:01:58.913651 22695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:01:58.914309 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22615
I20260928 15:01:58.914435 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:01:58.914621 2035 external_mini_cluster.cc:1428] Setting key 92f73531a1983da4b39af9476f619fc4
I20260928 15:01:58.915535 22695 catalog_manager.cc:1347] Generated new cluster ID: 55d18b2ed4534b738345d28a54e8341e
I20260928 15:01:58.915591 22695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:01:58 dist-test-slave-sf70 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:01:58.926796 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:58.915073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59915 (local address 127.0.0.1:46009)
0928 15:01:58.915615 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:58.915624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:58.915655 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:01:58.915777 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:58.915784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:58.915836 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:58.915991 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:58.916775 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.918204 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.919234 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.919846 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.919954 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:58.922921 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:58.922925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:58.922925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:58.922937 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:58.924896 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.925269 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.925272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.925272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.925371 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.925533 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.925534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.925534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.925812 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:58.925922 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:58.926395 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 15:01:58.926532 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":80,"threads_started":1}
I20260928 15:01:58.927685 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:01:58.936529 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:01:58.929206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52725 (local address 127.0.0.1:46009)
0928 15:01:58.929253 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:01:58.929254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:01:58.929387 (+ 133us) server_negotiation.cc:407] Connection header received
0928 15:01:58.929447 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:01:58.929447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:01:58.929470 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:01:58.929523 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:01:58.930165 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.930892 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.931932 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:01:58.932252 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:01:58.932309 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:01:58.933712 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:01:58.933714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:01:58.933714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:01:58.933720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:01:58.935208 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.935610 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.935611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.935612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.935680 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:01:58.935921 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:01:58.935922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:01:58.935922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:01:58.936050 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:01:58.936136 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:01:58.936276 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 15:01:58.936347 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 15:01:58.957827 22695 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:01:58.958617 22695 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:01:58.964578 22695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa7d184d1c3345898c902a39ffbf80e3: Generated new TSK 0
I20260928 15:01:58.964780 22695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:01:58.970121 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22615
2026-09-28T15:01:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:01:59 dist-test-slave-sf70 krb5kdc[22714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:01:59 dist-test-slave-sf70 krb5kdc[22714](info): set up 2 sockets
Sep 28 15:01:59 dist-test-slave-sf70 krb5kdc[22714](info): commencing operation
krb5kdc: starting...
W20260928 15:02:00.074543 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:00 dist-test-slave-sf70 krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:00.207996 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:00.292671 22727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:00.292924 22727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:00.292961 22727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:00.294132 22727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:00.294209 22727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:00.294235 22727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:00.294271 22727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:00.294301 22727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:00.296002 22727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:00.296276 22727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:00.296526 22727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:00.301566 22732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:00.301566 22735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:00.301575 22733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:00.302034 22727 server_base.cc:1034] running on GCE node
I20260928 15:02:00.302546 22727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:00.302887 22727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:00.304064 22727 hybrid_clock.cc:648] HybridClock initialized: now 1790607720304033 us; error 39 us; skew 500 ppm
Sep 28 15:02:00 dist-test-slave-sf70 krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:00.323062 22727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:00.323633 22727 webserver.cc:460] Webserver started at http://127.0.0.1:45017/ using document root <none> and password file <none>
I20260928 15:02:00.323973 22727 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:00.324033 22727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:00.324175 22727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:00.325287 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "bcbab564a98f463ca61b0a95f55682a6"
format_stamp: "Formatted at 2026-09-28 15:02:00 on dist-test-slave-sf70"
server_key: "465b9410eb008bedfb3ec2dd541b82ca"
server_key_iv: "db49b367294539bdf68a550db128af3c"
server_key_version: "encryptionkey@0"
I20260928 15:02:00.325642 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "bcbab564a98f463ca61b0a95f55682a6"
format_stamp: "Formatted at 2026-09-28 15:02:00 on dist-test-slave-sf70"
server_key: "465b9410eb008bedfb3ec2dd541b82ca"
server_key_iv: "db49b367294539bdf68a550db128af3c"
server_key_version: "encryptionkey@0"
I20260928 15:02:00.327075 22727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:00.327973 22742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:00.328315 22727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:00.328404 22727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "bcbab564a98f463ca61b0a95f55682a6"
format_stamp: "Formatted at 2026-09-28 15:02:00 on dist-test-slave-sf70"
server_key: "465b9410eb008bedfb3ec2dd541b82ca"
server_key_iv: "db49b367294539bdf68a550db128af3c"
server_key_version: "encryptionkey@0"
I20260928 15:02:00.328513 22727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:00.363685 22727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:00.366431 22727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:00.366609 22727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:00.371124 22727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33661
I20260928 15:02:00.371174 22795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33661 every 8 connection(s)
I20260928 15:02:00.371541 22727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:00.372677 22796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:00.374085 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22727
I20260928 15:02:00.374209 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:00.374446 2035 external_mini_cluster.cc:1428] Setting key 6c71be3ac12aa1c7d114e8f77e31a8e0
I20260928 15:02:00.375593 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6: Bootstrap starting.
I20260928 15:02:00.376331 22796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:00.376633 22796 log.cc:824] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:00.377447 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6: No bootstrap required, opened a new log
I20260928 15:02:00.380002 22796 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbab564a98f463ca61b0a95f55682a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33661 } }
I20260928 15:02:00.380138 22796 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:00.380162 22796 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbab564a98f463ca61b0a95f55682a6, State: Initialized, Role: FOLLOWER
I20260928 15:02:00.380326 22796 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbab564a98f463ca61b0a95f55682a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33661 } }
I20260928 15:02:00.380412 22796 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:00.380470 22796 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:00.380553 22796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:00.380775 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:00.375042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54363 (local address 127.0.0.1:33661)
0928 15:02:00.375223 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:00.375231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:00.375257 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:02:00.375752 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:00.375760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:00.375810 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:00.376003 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:00.376209 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.377370 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.378524 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.378948 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.379045 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:00.379875 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:00.379878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:00.379878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:00.380130 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:00.380192 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:00.380210 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:00.380290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:00.380370 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:02:00.380596 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 15:02:00.381229 22796 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbab564a98f463ca61b0a95f55682a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33661 } }
I20260928 15:02:00.381350 22796 leader_election.cc:302] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcbab564a98f463ca61b0a95f55682a6; no voters:
I20260928 15:02:00.381526 22796 leader_election.cc:288] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:00.381554 22801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:00.381690 22801 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [term 1 LEADER]: Becoming Leader. State: Replica: bcbab564a98f463ca61b0a95f55682a6, State: Running, Role: LEADER
I20260928 15:02:00.381691 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:00.381780 22801 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbab564a98f463ca61b0a95f55682a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33661 } }
I20260928 15:02:00.381814 22796 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:00.382104 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcbab564a98f463ca61b0a95f55682a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbab564a98f463ca61b0a95f55682a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33661 } } }
I20260928 15:02:00.382242 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:00.382418 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcbab564a98f463ca61b0a95f55682a6. Latest consensus state: current_term: 1 leader_uuid: "bcbab564a98f463ca61b0a95f55682a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbab564a98f463ca61b0a95f55682a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33661 } } }
I20260928 15:02:00.382534 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:00.382931 22807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:00.383442 22807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:00.385848 22807 catalog_manager.cc:1347] Generated new cluster ID: 4b38820c47144599ac95a8ca8b2fcddb
I20260928 15:02:00.385903 22807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:00.387519 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:00.383201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42451 (local address 127.0.0.1:33661)
0928 15:02:00.383246 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:00.383248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:00.383408 (+ 160us) server_negotiation.cc:407] Connection header received
0928 15:02:00.383505 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:00.383506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:00.383517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:00.383548 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:00.383876 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.385184 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.386204 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:00.386501 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:00.386575 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:00.387270 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:00.387271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:00.387271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:00.387276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:00.387290 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:00.387294 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:00.387344 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:00.387420 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:02:00.387445 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:00.393911 22807 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:00.394680 22807 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:00.401983 22807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcbab564a98f463ca61b0a95f55682a6: Generated new TSK 0
I20260928 15:02:00.402182 22807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:00.407253 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22727
2026-09-28T15:02:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:00 dist-test-slave-sf70 krb5kdc[22827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:00 dist-test-slave-sf70 krb5kdc[22827](info): set up 2 sockets
Sep 28 15:02:00 dist-test-slave-sf70 krb5kdc[22827](info): commencing operation
krb5kdc: starting...
W20260928 15:02:01.497759 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:01 dist-test-slave-sf70 krb5kdc[22827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:01.622534 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:01.705634 22841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:01.705857 22841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:01.705881 22841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:01.706964 22841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:01.707000 22841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:01.707015 22841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:01.707027 22841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:01.707039 22841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:01.708453 22841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:01.708657 22841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:01.708830 22841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:01.713825 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:01.713852 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:01.713829 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:01.714008 22841 server_base.cc:1034] running on GCE node
I20260928 15:02:01.714309 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:01.714643 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:01.715818 22841 hybrid_clock.cc:648] HybridClock initialized: now 1790607721715792 us; error 37 us; skew 500 ppm
Sep 28 15:02:01 dist-test-slave-sf70 krb5kdc[22827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:01.718323 22841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:01.718782 22841 webserver.cc:460] Webserver started at http://127.0.0.1:33631/ using document root <none> and password file <none>
I20260928 15:02:01.719014 22841 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:01.719082 22841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:01.719250 22841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:01.720392 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5"
format_stamp: "Formatted at 2026-09-28 15:02:01 on dist-test-slave-sf70"
server_key: "01bb5f6f2c83d367b4de3b2019549e99"
server_key_iv: "0daf7469f8e6e4993bce239a03949575"
server_key_version: "encryptionkey@0"
I20260928 15:02:01.720780 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5"
format_stamp: "Formatted at 2026-09-28 15:02:01 on dist-test-slave-sf70"
server_key: "01bb5f6f2c83d367b4de3b2019549e99"
server_key_iv: "0daf7469f8e6e4993bce239a03949575"
server_key_version: "encryptionkey@0"
I20260928 15:02:01.722146 22841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:01.722958 22856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:01.723176 22841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:01.723286 22841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5"
format_stamp: "Formatted at 2026-09-28 15:02:01 on dist-test-slave-sf70"
server_key: "01bb5f6f2c83d367b4de3b2019549e99"
server_key_iv: "0daf7469f8e6e4993bce239a03949575"
server_key_version: "encryptionkey@0"
I20260928 15:02:01.723363 22841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:01.726816 22841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:01.728947 22841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:01.729108 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:01.733168 22841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35733
I20260928 15:02:01.733225 22908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35733 every 8 connection(s)
I20260928 15:02:01.733592 22841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:01.734560 22909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:01.736901 22909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5: Bootstrap starting.
I20260928 15:02:01.737447 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22841
I20260928 15:02:01.737493 22909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:01.737670 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:01.737890 2035 external_mini_cluster.cc:1428] Setting key 2b91754506a9f94d9ef4110a337eb4b3
I20260928 15:02:01.737902 22909 log.cc:824] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:01.738765 22909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5: No bootstrap required, opened a new log
I20260928 15:02:01.740594 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:01.738462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59939 (local address 127.0.0.1:35733)
0928 15:02:01.738663 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:01.738672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:01.738705 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:02:01.739254 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:01.739263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:01.739297 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:01.739465 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:01.740142 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:01.740146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:01.740146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:01.740151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:01.740199 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:01.740213 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:01.740306 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:01.740340 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 15:02:01.740380 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260928 15:02:01.741678 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:01.741644 22909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260928 15:02:01.741830 22909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:01.741871 22909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a6ecbc744b428ebc7d8ce38f21fdf5, State: Initialized, Role: FOLLOWER
I20260928 15:02:01.741999 22909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260928 15:02:01.742093 22909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:01.742116 22909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:01.742182 22909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:01.742756 22909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260928 15:02:01.742867 22909 leader_election.cc:302] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a6ecbc744b428ebc7d8ce38f21fdf5; no voters:
I20260928 15:02:01.743063 22909 leader_election.cc:288] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:01.743103 22918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:01.743273 22918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [term 1 LEADER]: Becoming Leader. State: Replica: 35a6ecbc744b428ebc7d8ce38f21fdf5, State: Running, Role: LEADER
I20260928 15:02:01.743331 22909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:01.743427 22918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260928 15:02:01.743817 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a6ecbc744b428ebc7d8ce38f21fdf5. Latest consensus state: current_term: 1 leader_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } } }
I20260928 15:02:01.743827 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a6ecbc744b428ebc7d8ce38f21fdf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } } }
I20260928 15:02:01.744007 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:01.744094 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:01.744488 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:01.743351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33947 (local address 127.0.0.1:35733)
0928 15:02:01.743385 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:01.743387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:01.743396 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:01.743434 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:01.743435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:01.743439 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:01.743519 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:01.744196 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:01.744197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:01.744198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:01.744202 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:01.744216 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:01.744222 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:01.744279 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:01.744380 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:02:01.744392 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:01.744592 22928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:01.745152 22928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:01.749004 22928 catalog_manager.cc:1347] Generated new cluster ID: 122635a1fc6d48918c92c69b020496c4
I20260928 15:02:01.749065 22928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:01.763456 22928 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:01.763653 22928 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:01.788725 22928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35a6ecbc744b428ebc7d8ce38f21fdf5: Generated new TSK 0
I20260928 15:02:01.788965 22928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:01.794517 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22841
2026-09-28T15:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:01 dist-test-slave-sf70 krb5kdc[22940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:01 dist-test-slave-sf70 krb5kdc[22940](info): set up 2 sockets
Sep 28 15:02:01 dist-test-slave-sf70 krb5kdc[22940](info): commencing operation
krb5kdc: starting...
W20260928 15:02:02.837870 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:02 dist-test-slave-sf70 krb5kdc[22940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:02.967600 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:03.050256 22953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:03.050500 22953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:03.050549 22953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:03.051718 22953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:03.051786 22953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:03.051808 22953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:03.051829 22953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:03.051860 22953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:03.053363 22953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:03.053663 22953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:03.053897 22953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:03.058914 22959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:03.058946 22958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:03.058924 22961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:03.059386 22953 server_base.cc:1034] running on GCE node
I20260928 15:02:03.059654 22953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:03.059966 22953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:03.061118 22953 hybrid_clock.cc:648] HybridClock initialized: now 1790607723061089 us; error 42 us; skew 500 ppm
Sep 28 15:02:03 dist-test-slave-sf70 krb5kdc[22940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:03.063849 22953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:03.064397 22953 webserver.cc:460] Webserver started at http://127.0.0.1:41243/ using document root <none> and password file <none>
I20260928 15:02:03.064658 22953 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:03.064729 22953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:03.064881 22953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:03.065973 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "6f82526bfc36440cbf0bd0751ce40574"
format_stamp: "Formatted at 2026-09-28 15:02:03 on dist-test-slave-sf70"
server_key: "256f3bd0d12ab49053c7ae5f1241221c"
server_key_iv: "c9f8ecfb86d4b71efbf5d8508a90229f"
server_key_version: "encryptionkey@0"
I20260928 15:02:03.066344 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "6f82526bfc36440cbf0bd0751ce40574"
format_stamp: "Formatted at 2026-09-28 15:02:03 on dist-test-slave-sf70"
server_key: "256f3bd0d12ab49053c7ae5f1241221c"
server_key_iv: "c9f8ecfb86d4b71efbf5d8508a90229f"
server_key_version: "encryptionkey@0"
I20260928 15:02:03.067713 22953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:02:03.068565 22968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:03.068759 22953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:03.068859 22953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "6f82526bfc36440cbf0bd0751ce40574"
format_stamp: "Formatted at 2026-09-28 15:02:03 on dist-test-slave-sf70"
server_key: "256f3bd0d12ab49053c7ae5f1241221c"
server_key_iv: "c9f8ecfb86d4b71efbf5d8508a90229f"
server_key_version: "encryptionkey@0"
I20260928 15:02:03.068944 22953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:03.084309 22953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:03.086711 22953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:03.086881 22953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:03.091245 22953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36117
I20260928 15:02:03.091287 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36117 every 8 connection(s)
I20260928 15:02:03.091687 22953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:03.092767 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 22953
I20260928 15:02:03.092823 23022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:03.092900 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:03.093133 2035 external_mini_cluster.cc:1428] Setting key 0f4511fafb009eba79ed8475386b0836
I20260928 15:02:03.096406 23022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574: Bootstrap starting.
I20260928 15:02:03.097152 23022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:03.097466 23022 log.cc:824] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:03.098272 23022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574: No bootstrap required, opened a new log
Sep 28 15:02:03 dist-test-slave-sf70 krb5kdc[22940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:02:03.100842 23022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f82526bfc36440cbf0bd0751ce40574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260928 15:02:03.101007 23022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:03.101055 23022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f82526bfc36440cbf0bd0751ce40574, State: Initialized, Role: FOLLOWER
I20260928 15:02:03.101184 23022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f82526bfc36440cbf0bd0751ce40574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260928 15:02:03.101280 23022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:03.101325 23022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:03.101390 23022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:03.102203 23022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f82526bfc36440cbf0bd0751ce40574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260928 15:02:03.102317 23022 leader_election.cc:302] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f82526bfc36440cbf0bd0751ce40574; no voters:
I20260928 15:02:03.102511 23022 leader_election.cc:288] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:03.102605 23027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:03.102779 23027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [term 1 LEADER]: Becoming Leader. State: Replica: 6f82526bfc36440cbf0bd0751ce40574, State: Running, Role: LEADER
I20260928 15:02:03.102820 23022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:03.102903 23027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f82526bfc36440cbf0bd0751ce40574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260928 15:02:03.103380 23027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f82526bfc36440cbf0bd0751ce40574. Latest consensus state: current_term: 1 leader_uuid: "6f82526bfc36440cbf0bd0751ce40574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f82526bfc36440cbf0bd0751ce40574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } } }
I20260928 15:02:03.103484 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f82526bfc36440cbf0bd0751ce40574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f82526bfc36440cbf0bd0751ce40574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } } }
I20260928 15:02:03.103520 23027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:03.103649 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:03.103993 23036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:03.104745 23036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:03.105113 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:03.093682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:43879 (local address 127.0.0.1:36117)
0928 15:02:03.093870 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:03.093877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:03.093902 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:03.094391 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:03.094398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:03.094450 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:03.094572 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:03.095453 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:03.096427 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:03.097517 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:03.097970 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:03.098055 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:03.100849 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:03.100854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:03.100854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:03.100899 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:03.102989 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:03.103460 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:03.103464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:03.103464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:03.103590 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:03.103881 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:03.103898 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:03.103899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:03.104261 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:03.104486 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:03.104666 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 15:02:03.104773 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260928 15:02:03.106459 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:03.106679 23036 catalog_manager.cc:1347] Generated new cluster ID: 85f97cd2ce3c40d3a49eed5a2a540a0d
I20260928 15:02:03.106729 23036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:03.116418 23036 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:03.117107 23036 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:03.120685 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:03.108200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:40597 (local address 127.0.0.1:36117)
0928 15:02:03.108240 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:03.108242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:03.112745 (+ 4503us) server_negotiation.cc:407] Connection header received
0928 15:02:03.112846 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:03.112848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:03.112872 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:03.112967 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:03.113819 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:03.114892 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:03.115954 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:03.116285 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:03.116349 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:03.117849 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:03.117850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:03.117851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:03.117857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:03.119396 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:03.119826 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:03.119827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:03.119827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:03.119919 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:03.120144 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:03.120145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:03.120145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:03.120267 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:03.120359 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:03.120531 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:02:03.120596 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:02:03.123960 23036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f82526bfc36440cbf0bd0751ce40574: Generated new TSK 0
I20260928 15:02:03.124166 23036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:03.132521 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 22953
W20260928 15:02:03.137133 22938 connection.cc:535] client connection to 127.0.0.1:36117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36117): Connection reset by peer (error 104)
2026-09-28T15:02:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:03 dist-test-slave-sf70 krb5kdc[23053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:03 dist-test-slave-sf70 krb5kdc[23053](info): set up 2 sockets
Sep 28 15:02:03 dist-test-slave-sf70 krb5kdc[23053](info): commencing operation
krb5kdc: starting...
W20260928 15:02:04.161684 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:04 dist-test-slave-sf70 krb5kdc[23053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:04.284790 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:04.367765 23066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:04.368018 23066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:04.368042 23066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:04.369310 23066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:04.369356 23066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:04.369371 23066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:04.369383 23066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:04.369436 23066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:04.370954 23066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:04.371214 23066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:04.371459 23066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:04.376454 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:04.376459 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:04.376467 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:04.376650 23066 server_base.cc:1034] running on GCE node
I20260928 15:02:04.377014 23066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:04.377405 23066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:04.378598 23066 hybrid_clock.cc:648] HybridClock initialized: now 1790607724378573 us; error 44 us; skew 500 ppm
Sep 28 15:02:04 dist-test-slave-sf70 krb5kdc[23053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:04.381179 23066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:04.381675 23066 webserver.cc:460] Webserver started at http://127.0.0.1:39451/ using document root <none> and password file <none>
I20260928 15:02:04.381908 23066 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:04.381976 23066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:04.382138 23066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:04.383241 23066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "c1c70c58b2fa4d63a2c3fb8deec43409"
format_stamp: "Formatted at 2026-09-28 15:02:04 on dist-test-slave-sf70"
server_key: "f290b66d6c8273c8d4e3c6a347bd0bfe"
server_key_iv: "9b0971c24ba513b7292c4a529ca2d251"
server_key_version: "encryptionkey@0"
I20260928 15:02:04.383610 23066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "c1c70c58b2fa4d63a2c3fb8deec43409"
format_stamp: "Formatted at 2026-09-28 15:02:04 on dist-test-slave-sf70"
server_key: "f290b66d6c8273c8d4e3c6a347bd0bfe"
server_key_iv: "9b0971c24ba513b7292c4a529ca2d251"
server_key_version: "encryptionkey@0"
I20260928 15:02:04.385082 23066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:04.385885 23081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:04.386121 23066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:04.386245 23066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "c1c70c58b2fa4d63a2c3fb8deec43409"
format_stamp: "Formatted at 2026-09-28 15:02:04 on dist-test-slave-sf70"
server_key: "f290b66d6c8273c8d4e3c6a347bd0bfe"
server_key_iv: "9b0971c24ba513b7292c4a529ca2d251"
server_key_version: "encryptionkey@0"
I20260928 15:02:04.386351 23066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:04.401423 23066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:04.403502 23066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:04.403671 23066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:04.407936 23066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46013
I20260928 15:02:04.407960 23134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46013 every 8 connection(s)
I20260928 15:02:04.408334 23066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:04.409360 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:04.409739 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23066
I20260928 15:02:04.409868 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:04.410049 2035 external_mini_cluster.cc:1428] Setting key d8ba9c4746a859e2fec9ec896d9721d4
I20260928 15:02:04.412424 23135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409: Bootstrap starting.
I20260928 15:02:04.413081 23135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:04.413410 23135 log.cc:824] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:04.414132 23135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409: No bootstrap required, opened a new log
I20260928 15:02:04.416109 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:04.410592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:54147 (local address 127.0.0.1:46013)
0928 15:02:04.410783 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:04.410790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:04.410816 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:02:04.411296 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:04.411303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:04.411354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:04.411483 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:04.411722 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.412936 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.413997 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.414491 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.414576 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:04.415317 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:04.415321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:04.415321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:04.415530 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:04.415562 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:04.415573 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:04.415646 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:04.415784 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 15:02:04.415837 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 15:02:04.416492 23135 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260928 15:02:04.416666 23135 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:04.416723 23135 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c70c58b2fa4d63a2c3fb8deec43409, State: Initialized, Role: FOLLOWER
I20260928 15:02:04.416904 23135 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260928 15:02:04.417008 23135 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:04.417047 23135 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:04.417106 23135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:04.417367 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:04.417807 23135 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260928 15:02:04.417917 23135 leader_election.cc:302] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c70c58b2fa4d63a2c3fb8deec43409; no voters:
I20260928 15:02:04.418099 23135 leader_election.cc:288] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:04.418174 23140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:04.418323 23140 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [term 1 LEADER]: Becoming Leader. State: Replica: c1c70c58b2fa4d63a2c3fb8deec43409, State: Running, Role: LEADER
I20260928 15:02:04.418378 23135 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:04.418512 23140 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260928 15:02:04.419770 23142 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } } }
I20260928 15:02:04.419866 23142 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:04.420113 23141 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c70c58b2fa4d63a2c3fb8deec43409. Latest consensus state: current_term: 1 leader_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c70c58b2fa4d63a2c3fb8deec43409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } } }
W20260928 15:02:04.420719 23160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 15:02:04.421128 23141 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:04.421363 23160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:04.421422 23156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:04.421627 23156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:04.422637 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:04.418910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:54619 (local address 127.0.0.1:46013)
0928 15:02:04.418944 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:04.418947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:04.419050 (+ 103us) server_negotiation.cc:407] Connection header received
0928 15:02:04.419123 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:04.419123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:04.419131 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:04.419208 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:04.419407 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.420088 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.421102 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:04.421446 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:04.421586 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:04.422323 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:04.422324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:04.422325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:04.422329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:04.422344 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:04.422349 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:04.422394 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:04.422542 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 15:02:04.422552 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 15:02:04.423249 23156 catalog_manager.cc:1347] Generated new cluster ID: df7e1772bac1401b91f18225749c2f1f
I20260928 15:02:04.423305 23156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:04.449525 23156 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:04.450111 23156 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:04.456974 23156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c70c58b2fa4d63a2c3fb8deec43409: Generated new TSK 0
I20260928 15:02:04.457204 23156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:04.463740 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23066
W20260928 15:02:04.468731 23051 connection.cc:535] client connection to 127.0.0.1:46013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46013): Connection reset by peer (error 104)
2026-09-28T15:02:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:04 dist-test-slave-sf70 krb5kdc[23166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:04 dist-test-slave-sf70 krb5kdc[23166](info): set up 2 sockets
Sep 28 15:02:04 dist-test-slave-sf70 krb5kdc[23166](info): commencing operation
krb5kdc: starting...
W20260928 15:02:05.549688 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:05 dist-test-slave-sf70 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:05.671432 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:05.752111 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:05.752311 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:05.752333 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:05.753533 23179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:05.753578 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:05.753593 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:05.753605 23179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:05.753618 23179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:05.755323 23179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:05.755577 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:05.755820 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:05.760841 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:05.760848 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:05.760842 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:05.761080 23179 server_base.cc:1034] running on GCE node
I20260928 15:02:05.761368 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:05.761606 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:05.762775 23179 hybrid_clock.cc:648] HybridClock initialized: now 1790607725762762 us; error 29 us; skew 500 ppm
Sep 28 15:02:05 dist-test-slave-sf70 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:05.765292 23179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:05.765676 23179 webserver.cc:460] Webserver started at http://127.0.0.1:42303/ using document root <none> and password file <none>
I20260928 15:02:05.765872 23179 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:05.765923 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:05.766008 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:05.767046 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "e0c1165f6011434badf9c06492103f65"
format_stamp: "Formatted at 2026-09-28 15:02:05 on dist-test-slave-sf70"
server_key: "25f5e6601278397ad736e41a1673da48"
server_key_iv: "ed60a5c22061791a3dcdaf2f6ecc4237"
server_key_version: "encryptionkey@0"
I20260928 15:02:05.767355 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "e0c1165f6011434badf9c06492103f65"
format_stamp: "Formatted at 2026-09-28 15:02:05 on dist-test-slave-sf70"
server_key: "25f5e6601278397ad736e41a1673da48"
server_key_iv: "ed60a5c22061791a3dcdaf2f6ecc4237"
server_key_version: "encryptionkey@0"
I20260928 15:02:05.768927 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:05.770231 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:05.770439 23179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:02:05.770608 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "e0c1165f6011434badf9c06492103f65"
format_stamp: "Formatted at 2026-09-28 15:02:05 on dist-test-slave-sf70"
server_key: "25f5e6601278397ad736e41a1673da48"
server_key_iv: "ed60a5c22061791a3dcdaf2f6ecc4237"
server_key_version: "encryptionkey@0"
I20260928 15:02:05.770684 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:05.774362 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:05.776440 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:05.776556 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:05.780611 23179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46525
I20260928 15:02:05.780640 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46525 every 8 connection(s)
I20260928 15:02:05.781013 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:05.782037 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:05.784394 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65: Bootstrap starting.
I20260928 15:02:05.784986 23247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:05.785266 23247 log.cc:824] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:05.786067 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65: No bootstrap required, opened a new log
I20260928 15:02:05.786072 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23179
I20260928 15:02:05.786216 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:05.786448 2035 external_mini_cluster.cc:1428] Setting key 0fdfcc4a38521350fd1cce303c59f062
I20260928 15:02:05.788098 23247 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c1165f6011434badf9c06492103f65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260928 15:02:05.788256 23247 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:05.788336 23247 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c1165f6011434badf9c06492103f65, State: Initialized, Role: FOLLOWER
I20260928 15:02:05.788514 23247 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c1165f6011434badf9c06492103f65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260928 15:02:05.788612 23247 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:05.788643 23247 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:05.788720 23247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:05.789553 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:05.787024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:34021 (local address 127.0.0.1:46525)
0928 15:02:05.787228 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:05.787238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:05.787272 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:02:05.788090 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:05.788098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:05.788130 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:05.788295 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:05.789035 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:05.789038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:05.789039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:05.789044 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:05.789107 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:05.789121 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:05.789209 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:05.789287 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:02:05.789338 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 15:02:05.789868 23247 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c1165f6011434badf9c06492103f65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260928 15:02:05.789987 23247 leader_election.cc:302] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c1165f6011434badf9c06492103f65; no voters:
I20260928 15:02:05.790161 23247 leader_election.cc:288] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:05.790194 23252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:05.790295 23252 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [term 1 LEADER]: Becoming Leader. State: Replica: e0c1165f6011434badf9c06492103f65, State: Running, Role: LEADER
I20260928 15:02:05.790372 23252 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c1165f6011434badf9c06492103f65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260928 15:02:05.790405 23247 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:05.790545 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:05.790691 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c1165f6011434badf9c06492103f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c1165f6011434badf9c06492103f65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } } }
I20260928 15:02:05.790767 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:05.790951 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c1165f6011434badf9c06492103f65. Latest consensus state: current_term: 1 leader_uuid: "e0c1165f6011434badf9c06492103f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c1165f6011434badf9c06492103f65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } } }
I20260928 15:02:05.791041 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:05.791173 23258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:05.791688 23258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:05.793579 23258 catalog_manager.cc:1347] Generated new cluster ID: acdefb298bbd40c2900fcc63a1993c14
I20260928 15:02:05.793637 23258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:05.801448 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:05.800212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:52053 (local address 127.0.0.1:46525)
0928 15:02:05.800250 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:05.800252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:05.800403 (+ 151us) server_negotiation.cc:407] Connection header received
0928 15:02:05.800487 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:05.800488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:05.800494 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:05.800584 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:05.801221 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:05.801222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:05.801223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:05.801227 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:05.801243 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:05.801247 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:05.801305 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:05.801361 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 15:02:05.801377 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:02:05.809448 23258 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:05.809612 23258 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:05.820096 23258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c1165f6011434badf9c06492103f65: Generated new TSK 0
I20260928 15:02:05.820319 23258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:05.823065 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23179
2026-09-28T15:02:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:05 dist-test-slave-sf70 krb5kdc[23279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:05 dist-test-slave-sf70 krb5kdc[23279](info): set up 2 sockets
Sep 28 15:02:05 dist-test-slave-sf70 krb5kdc[23279](info): commencing operation
krb5kdc: starting...
W20260928 15:02:06.885727 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:06 dist-test-slave-sf70 krb5kdc[23279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:07.008050 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:07.087947 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:07.088213 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:07.088251 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:07.089402 23292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:07.089475 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:07.089499 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:07.089525 23292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:07.089557 23292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:07.090992 23292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:07.091266 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:07.091495 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:07.097077 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:07.097085 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:07.097077 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:07.097517 23292 server_base.cc:1034] running on GCE node
I20260928 15:02:07.097730 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:07.097981 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:07.099145 23292 hybrid_clock.cc:648] HybridClock initialized: now 1790607727099129 us; error 29 us; skew 500 ppm
Sep 28 15:02:07 dist-test-slave-sf70 krb5kdc[23279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:07.101738 23292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:07.102289 23292 webserver.cc:460] Webserver started at http://127.0.0.1:43539/ using document root <none> and password file <none>
I20260928 15:02:07.102537 23292 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:07.102588 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:07.102775 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:07.103982 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "e375f5d7b3c44fc5b862f649ff1e72f1"
format_stamp: "Formatted at 2026-09-28 15:02:07 on dist-test-slave-sf70"
server_key: "b86a444ae853c1c962000486f76682a4"
server_key_iv: "b0535e1e0f135750b71b56209846733d"
server_key_version: "encryptionkey@0"
I20260928 15:02:07.104410 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "e375f5d7b3c44fc5b862f649ff1e72f1"
format_stamp: "Formatted at 2026-09-28 15:02:07 on dist-test-slave-sf70"
server_key: "b86a444ae853c1c962000486f76682a4"
server_key_iv: "b0535e1e0f135750b71b56209846733d"
server_key_version: "encryptionkey@0"
I20260928 15:02:07.105765 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:02:07.106741 23307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:07.106992 23292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:07.107118 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "e375f5d7b3c44fc5b862f649ff1e72f1"
format_stamp: "Formatted at 2026-09-28 15:02:07 on dist-test-slave-sf70"
server_key: "b86a444ae853c1c962000486f76682a4"
server_key_iv: "b0535e1e0f135750b71b56209846733d"
server_key_version: "encryptionkey@0"
I20260928 15:02:07.107203 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:07.139155 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:07.141433 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:07.141618 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:07.146276 23292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33497
I20260928 15:02:07.146317 23359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33497 every 8 connection(s)
I20260928 15:02:07.146704 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:07.147698 23360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:07.149996 23360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1: Bootstrap starting.
I20260928 15:02:07.150537 23360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:07.150770 23360 log.cc:824] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:07.151458 23360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1: No bootstrap required, opened a new log
I20260928 15:02:07.152798 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23292
I20260928 15:02:07.152937 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:07.153115 2035 external_mini_cluster.cc:1428] Setting key 92406e60c279ebe3482a2eacdd4ca88e
I20260928 15:02:07.153170 23360 raft_consensus.cc:348] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33497 } }
I20260928 15:02:07.153383 23360 raft_consensus.cc:374] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:07.153425 23360 raft_consensus.cc:729] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e375f5d7b3c44fc5b862f649ff1e72f1, State: Initialized, Role: FOLLOWER
I20260928 15:02:07.153599 23360 consensus_queue.cc:260] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33497 } }
I20260928 15:02:07.153697 23360 raft_consensus.cc:388] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:07.153741 23360 raft_consensus.cc:482] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:07.153795 23360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:07.154500 23360 raft_consensus.cc:504] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33497 } }
I20260928 15:02:07.154623 23360 leader_election.cc:302] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e375f5d7b3c44fc5b862f649ff1e72f1; no voters:
I20260928 15:02:07.154773 23360 leader_election.cc:288] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:07.154793 23366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:07.154904 23366 raft_consensus.cc:686] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [term 1 LEADER]: Becoming Leader. State: Replica: e375f5d7b3c44fc5b862f649ff1e72f1, State: Running, Role: LEADER
I20260928 15:02:07.154994 23366 consensus_queue.cc:237] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33497 } }
I20260928 15:02:07.155032 23360 sys_catalog.cc:564] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:07.155347 23367 sys_catalog.cc:455] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33497 } } }
I20260928 15:02:07.155614 23367 sys_catalog.cc:458] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:07.155531 23368 sys_catalog.cc:455] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e375f5d7b3c44fc5b862f649ff1e72f1. Latest consensus state: current_term: 1 leader_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e375f5d7b3c44fc5b862f649ff1e72f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33497 } } }
I20260928 15:02:07.155870 23368 sys_catalog.cc:458] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:07.156059 23375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:07.156543 23375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:07.158197 23375 catalog_manager.cc:1347] Generated new cluster ID: aae7c22a8b7945789089112814dd6aa8
I20260928 15:02:07.158247 23375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:02:07 dist-test-slave-sf70 krb5kdc[23279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:02:07.165760 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:07.153666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:49723 (local address 127.0.0.1:33497)
0928 15:02:07.153942 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:07.153952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:07.153984 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:02:07.154399 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:07.154408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:07.154474 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:07.154627 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:07.155485 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:07.156985 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:07.158025 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:07.158612 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:07.158709 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:07.161560 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:07.161564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:07.161564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:07.161577 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:07.164001 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:07.164439 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:07.164442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:07.164442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:07.164564 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:07.164796 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:07.164797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:07.164797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:07.165149 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:07.165316 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:07.165428 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 15:02:07.165531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260928 15:02:07.166846 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:07.168140 23375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:07.168676 23375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:07.171948 23375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e375f5d7b3c44fc5b862f649ff1e72f1: Generated new TSK 0
I20260928 15:02:07.172113 23375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:07.178658 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:07.168336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:47755 (local address 127.0.0.1:33497)
0928 15:02:07.168370 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:07.168371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:07.171645 (+ 3274us) server_negotiation.cc:407] Connection header received
0928 15:02:07.171856 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:07.171857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:07.171869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:07.171951 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:07.172629 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:07.173449 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:07.174118 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:07.174450 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:07.174525 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:07.175826 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:07.175828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:07.175828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:07.175835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:07.177435 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:07.177833 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:07.177834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:07.177834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:07.177899 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:07.178111 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:07.178112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:07.178112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:07.178241 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:07.178356 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:07.178466 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 15:02:07.178543 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:07.180799 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23292
W20260928 15:02:07.185194 23277 connection.cc:535] client connection to 127.0.0.1:33497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33497): Connection reset by peer (error 104)
2026-09-28T15:02:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:07 dist-test-slave-sf70 krb5kdc[23392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:07 dist-test-slave-sf70 krb5kdc[23392](info): set up 2 sockets
Sep 28 15:02:07 dist-test-slave-sf70 krb5kdc[23392](info): commencing operation
krb5kdc: starting...
W20260928 15:02:08.225780 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:08 dist-test-slave-sf70 krb5kdc[23392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:08.355790 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:08.437561 23405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:08.437769 23405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:08.437793 23405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:08.439009 23405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:08.439054 23405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:08.439069 23405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:08.439082 23405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:08.439095 23405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:08.440639 23405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:08.440896 23405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:08.441128 23405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:08.446818 23411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:08.446823 23410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:08.446918 23405 server_base.cc:1034] running on GCE node
W20260928 15:02:08.446817 23413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:08.447332 23405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:08.447646 23405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:08.448796 23405 hybrid_clock.cc:648] HybridClock initialized: now 1790607728448766 us; error 45 us; skew 500 ppm
Sep 28 15:02:08 dist-test-slave-sf70 krb5kdc[23392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:08.451272 23405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:08.451695 23405 webserver.cc:460] Webserver started at http://127.0.0.1:35511/ using document root <none> and password file <none>
I20260928 15:02:08.451962 23405 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:08.452013 23405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:08.452105 23405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:08.453145 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "15fed7fa1bc842dbb1d3bf819946366f"
format_stamp: "Formatted at 2026-09-28 15:02:08 on dist-test-slave-sf70"
server_key: "2992613e8f6f6698f9d9ec69debbe62e"
server_key_iv: "d758a88373c309f2d261fd842c4342db"
server_key_version: "encryptionkey@0"
I20260928 15:02:08.453464 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "15fed7fa1bc842dbb1d3bf819946366f"
format_stamp: "Formatted at 2026-09-28 15:02:08 on dist-test-slave-sf70"
server_key: "2992613e8f6f6698f9d9ec69debbe62e"
server_key_iv: "d758a88373c309f2d261fd842c4342db"
server_key_version: "encryptionkey@0"
I20260928 15:02:08.454746 23405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:02:08.455492 23420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:08.455760 23405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:08.455842 23405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "15fed7fa1bc842dbb1d3bf819946366f"
format_stamp: "Formatted at 2026-09-28 15:02:08 on dist-test-slave-sf70"
server_key: "2992613e8f6f6698f9d9ec69debbe62e"
server_key_iv: "d758a88373c309f2d261fd842c4342db"
server_key_version: "encryptionkey@0"
I20260928 15:02:08.455946 23405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:08.464931 23405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:08.466879 23405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:08.466989 23405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:08.471293 23405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36081
I20260928 15:02:08.471333 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36081 every 8 connection(s)
I20260928 15:02:08.471665 23405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:08.472541 23473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:08.474830 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f: Bootstrap starting.
I20260928 15:02:08.475409 23473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:08.475708 23473 log.cc:824] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:08.476547 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f: No bootstrap required, opened a new log
I20260928 15:02:08.478185 23473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fed7fa1bc842dbb1d3bf819946366f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260928 15:02:08.478364 23473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:08.478418 23473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fed7fa1bc842dbb1d3bf819946366f, State: Initialized, Role: FOLLOWER
I20260928 15:02:08.478577 23473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fed7fa1bc842dbb1d3bf819946366f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260928 15:02:08.478680 23473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:08.478724 23473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:08.478778 23473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:08.479316 23473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fed7fa1bc842dbb1d3bf819946366f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260928 15:02:08.479441 23473 leader_election.cc:302] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15fed7fa1bc842dbb1d3bf819946366f; no voters:
I20260928 15:02:08.479627 23473 leader_election.cc:288] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:08.479679 23476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:08.479853 23476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [term 1 LEADER]: Becoming Leader. State: Replica: 15fed7fa1bc842dbb1d3bf819946366f, State: Running, Role: LEADER
I20260928 15:02:08.479912 23473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:08.479967 23476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fed7fa1bc842dbb1d3bf819946366f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260928 15:02:08.480365 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15fed7fa1bc842dbb1d3bf819946366f. Latest consensus state: current_term: 1 leader_uuid: "15fed7fa1bc842dbb1d3bf819946366f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fed7fa1bc842dbb1d3bf819946366f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } } }
I20260928 15:02:08.480511 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:08.480492 23477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15fed7fa1bc842dbb1d3bf819946366f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fed7fa1bc842dbb1d3bf819946366f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } } }
I20260928 15:02:08.480638 23477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:08.480805 23485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:08.480849 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23405
I20260928 15:02:08.480974 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:08.481225 2035 external_mini_cluster.cc:1428] Setting key 03b84b14a5454cb2d3f3c643f491cc04
I20260928 15:02:08.481356 23485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:08.483731 23485 catalog_manager.cc:1347] Generated new cluster ID: 0c4826a181b74fd08f706d2db51c65b9
I20260928 15:02:08.483785 23485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:08.486817 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:08.481944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:48813 (local address 127.0.0.1:36081)
0928 15:02:08.482278 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:08.482287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:08.482319 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:02:08.482430 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:08.482437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:08.482505 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:08.482653 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:08.482873 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:08.484125 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:08.484911 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:08.485328 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:08.485443 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:08.486148 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:08.486152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:08.486152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:08.486323 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:08.486354 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:08.486365 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:08.486455 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:08.486534 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 15:02:08.486656 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260928 15:02:08.487735 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:08.494937 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:08.489114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:48571 (local address 127.0.0.1:36081)
0928 15:02:08.489155 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:08.489156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:08.492103 (+ 2947us) server_negotiation.cc:407] Connection header received
0928 15:02:08.492191 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:08.492192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:08.492206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:08.492254 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:08.492424 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:08.492912 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:08.493668 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:08.493901 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:08.493977 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:08.494681 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:08.494682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:08.494683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:08.494687 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:08.494702 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:08.494706 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:08.494758 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:08.494826 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 15:02:08.494837 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:08.504266 23485 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:08.504791 23485 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:08.518707 23485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15fed7fa1bc842dbb1d3bf819946366f: Generated new TSK 0
I20260928 15:02:08.518913 23485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:08.521862 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23405
W20260928 15:02:08.526444 23390 connection.cc:535] client connection to 127.0.0.1:36081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36081): Connection reset by peer (error 104)
2026-09-28T15:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:08 dist-test-slave-sf70 krb5kdc[23505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:08 dist-test-slave-sf70 krb5kdc[23505](info): set up 2 sockets
Sep 28 15:02:08 dist-test-slave-sf70 krb5kdc[23505](info): commencing operation
krb5kdc: starting...
W20260928 15:02:09.581619 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:09 dist-test-slave-sf70 krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:02:09.708217 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:09.790721 23518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:09.790930 23518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:09.790952 23518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:09.792121 23518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:09.792166 23518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:09.792181 23518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:09.792193 23518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:09.792244 23518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:09.793713 23518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:09.793967 23518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:09.794190 23518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:09.799180 23524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:09.799206 23523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:09.799187 23526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:09.799217 23518 server_base.cc:1034] running on GCE node
I20260928 15:02:09.799695 23518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:09.800042 23518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:09.801213 23518 hybrid_clock.cc:648] HybridClock initialized: now 1790607729801200 us; error 30 us; skew 500 ppm
Sep 28 15:02:09 dist-test-slave-sf70 krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:09.803746 23518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:02:09.804253 23518 webserver.cc:460] Webserver started at http://127.0.0.1:40119/ using document root <none> and password file <none>
I20260928 15:02:09.804505 23518 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:09.804571 23518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:09.804711 23518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:09.805804 23518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "4227da31ce99447fba72941b27f27e55"
format_stamp: "Formatted at 2026-09-28 15:02:09 on dist-test-slave-sf70"
server_key: "797f2b5ca07073e65161c05775502bdb"
server_key_iv: "79c6131165649dc3ac8fc341d370e522"
server_key_version: "encryptionkey@0"
I20260928 15:02:09.806159 23518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "4227da31ce99447fba72941b27f27e55"
format_stamp: "Formatted at 2026-09-28 15:02:09 on dist-test-slave-sf70"
server_key: "797f2b5ca07073e65161c05775502bdb"
server_key_iv: "79c6131165649dc3ac8fc341d370e522"
server_key_version: "encryptionkey@0"
I20260928 15:02:09.807523 23518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:09.808422 23533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:09.808624 23518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:09.808723 23518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "4227da31ce99447fba72941b27f27e55"
format_stamp: "Formatted at 2026-09-28 15:02:09 on dist-test-slave-sf70"
server_key: "797f2b5ca07073e65161c05775502bdb"
server_key_iv: "79c6131165649dc3ac8fc341d370e522"
server_key_version: "encryptionkey@0"
I20260928 15:02:09.808801 23518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:09.812180 23518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:09.814124 23518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:09.814283 23518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:09.818372 23518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44625
I20260928 15:02:09.818400 23585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44625 every 8 connection(s)
I20260928 15:02:09.818753 23518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:09.819718 23586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:09.821983 23586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55: Bootstrap starting.
I20260928 15:02:09.822512 23586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:09.822746 23586 log.cc:824] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:09.823036 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23518
I20260928 15:02:09.823128 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:09.823295 2035 external_mini_cluster.cc:1428] Setting key 535501768a5a59cc7b4bea7d5f7a01f1
I20260928 15:02:09.823840 23586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55: No bootstrap required, opened a new log
I20260928 15:02:09.825980 23586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4227da31ce99447fba72941b27f27e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260928 15:02:09.826177 23586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:09.826220 23586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4227da31ce99447fba72941b27f27e55, State: Initialized, Role: FOLLOWER
I20260928 15:02:09.826371 23586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4227da31ce99447fba72941b27f27e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260928 15:02:09.826512 23586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:09.826547 23586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:09.826601 23586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:09.826632 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:09.823798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:34689 (local address 127.0.0.1:44625)
0928 15:02:09.824048 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:09.824056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:09.824090 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:02:09.825106 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:09.825115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:09.825149 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:09.825297 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:09.825983 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:09.825987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:09.825987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:09.825992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:09.826036 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:09.826050 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:09.826115 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:09.826192 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 15:02:09.826238 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260928 15:02:09.827365 23586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4227da31ce99447fba72941b27f27e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260928 15:02:09.827419 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:09.827476 23586 leader_election.cc:302] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4227da31ce99447fba72941b27f27e55; no voters:
I20260928 15:02:09.827623 23586 leader_election.cc:288] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:09.827685 23591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:09.827847 23586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:09.827880 23591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [term 1 LEADER]: Becoming Leader. State: Replica: 4227da31ce99447fba72941b27f27e55, State: Running, Role: LEADER
I20260928 15:02:09.828028 23591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4227da31ce99447fba72941b27f27e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260928 15:02:09.828339 23591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4227da31ce99447fba72941b27f27e55. Latest consensus state: current_term: 1 leader_uuid: "4227da31ce99447fba72941b27f27e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4227da31ce99447fba72941b27f27e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } } }
I20260928 15:02:09.828349 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4227da31ce99447fba72941b27f27e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4227da31ce99447fba72941b27f27e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } } }
I20260928 15:02:09.828502 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:09.828708 23591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:09.829116 23602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:09.829950 23602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:09.830233 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:09.828996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:49015 (local address 127.0.0.1:44625)
0928 15:02:09.829029 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:09.829031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:09.829202 (+ 171us) server_negotiation.cc:407] Connection header received
0928 15:02:09.829292 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:09.829293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:09.829297 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:09.829360 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:09.829989 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:09.829991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:09.829991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:09.829995 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:09.830007 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:09.830011 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:09.830061 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:09.830152 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:02:09.830167 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:09.831462 23602 catalog_manager.cc:1347] Generated new cluster ID: b764887ea9e54c60907031921cb8bbb9
I20260928 15:02:09.831512 23602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:09.842901 23602 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:09.843087 23602 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:09.849897 23602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4227da31ce99447fba72941b27f27e55: Generated new TSK 0
I20260928 15:02:09.850091 23602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:09.858033 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23518
2026-09-28T15:02:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:09 dist-test-slave-sf70 krb5kdc[23617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:09 dist-test-slave-sf70 krb5kdc[23617](info): set up 2 sockets
Sep 28 15:02:09 dist-test-slave-sf70 krb5kdc[23617](info): commencing operation
krb5kdc: starting...
W20260928 15:02:10.897661 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:10 dist-test-slave-sf70 krb5kdc[23617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:11.019054 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:34021
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:34021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:11.101756 23630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:11.102030 23630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:11.102070 23630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:11.103171 23630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:11.103214 23630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:11.103230 23630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:11.103245 23630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:11.103259 23630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:11.104883 23630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:34021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:11.105091 23630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:11.105269 23630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:11.110219 23637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:11.110193 23639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:11.110391 23630 server_base.cc:1034] running on GCE node
W20260928 15:02:11.110220 23636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:11.110744 23630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:11.111016 23630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:11.112185 23630 hybrid_clock.cc:648] HybridClock initialized: now 1790607731112146 us; error 77 us; skew 500 ppm
Sep 28 15:02:11 dist-test-slave-sf70 krb5kdc[23617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607731, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:11.119827 23630 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:11.120385 23630 webserver.cc:460] Webserver started at http://127.1.252.254:40121/ using document root <none> and password file <none>
I20260928 15:02:11.120625 23630 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:11.120671 23630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:11.120783 23630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:11.121891 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "e37ed06dcf8f4d4588e942c3655bfa1c"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-sf70"
server_key: "99d448539d41f6154eabdf8a000fae54"
server_key_iv: "ba86b9adc6164ed164d8cd688db7081c"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.122225 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "e37ed06dcf8f4d4588e942c3655bfa1c"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-sf70"
server_key: "99d448539d41f6154eabdf8a000fae54"
server_key_iv: "ba86b9adc6164ed164d8cd688db7081c"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.123631 23630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:02:11.124511 23646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:11.124761 23630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:11.124859 23630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "e37ed06dcf8f4d4588e942c3655bfa1c"
format_stamp: "Formatted at 2026-09-28 15:02:11 on dist-test-slave-sf70"
server_key: "99d448539d41f6154eabdf8a000fae54"
server_key_iv: "ba86b9adc6164ed164d8cd688db7081c"
server_key_version: "encryptionkey@0"
I20260928 15:02:11.124910 23630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:11.145319 23630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:11.147536 23630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:11.147648 23630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:11.151634 23630 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:34021
I20260928 15:02:11.151705 23698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:34021 every 8 connection(s)
I20260928 15:02:11.152060 23630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:11.152969 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:11.153762 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23630
I20260928 15:02:11.153899 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:11.154109 2035 external_mini_cluster.cc:1428] Setting key b3fe6279b76bdc3f6481f5a02a25847e
I20260928 15:02:11.156365 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c: Bootstrap starting.
I20260928 15:02:11.156984 23699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:11.157287 23699 log.cc:824] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:11.158043 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c: No bootstrap required, opened a new log
I20260928 15:02:11.160979 23699 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34021 } }
I20260928 15:02:11.161150 23699 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:11.161201 23699 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37ed06dcf8f4d4588e942c3655bfa1c, State: Initialized, Role: FOLLOWER
I20260928 15:02:11.161375 23699 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34021 } }
Sep 28 15:02:11 dist-test-slave-sf70 krb5kdc[23617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:11.161480 23699 raft_consensus.cc:388] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:11.161530 23699 raft_consensus.cc:482] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:11.161592 23699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:11.162354 23699 raft_consensus.cc:504] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34021 } }
I20260928 15:02:11.162493 23699 leader_election.cc:302] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e37ed06dcf8f4d4588e942c3655bfa1c; no voters:
I20260928 15:02:11.162701 23699 leader_election.cc:288] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:11.162736 23704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:11.162850 23704 raft_consensus.cc:686] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [term 1 LEADER]: Becoming Leader. State: Replica: e37ed06dcf8f4d4588e942c3655bfa1c, State: Running, Role: LEADER
I20260928 15:02:11.163004 23699 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:11.163007 23704 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34021 } }
I20260928 15:02:11.163455 23706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37ed06dcf8f4d4588e942c3655bfa1c. Latest consensus state: current_term: 1 leader_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34021 } } }
I20260928 15:02:11.163604 23706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:11.164129 23712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:11.164697 23712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:11.164697 23705 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ed06dcf8f4d4588e942c3655bfa1c" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 34021 } } }
I20260928 15:02:11.165735 23705 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:11.166067 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.154631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56207 (local address 127.1.252.254:34021)
0928 15:02:11.154814 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.154822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.154847 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:02:11.155240 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.155246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.155290 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.155403 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.156358 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.157404 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.158515 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.158941 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.159027 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.162118 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.162122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.162123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.162133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.164312 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.164808 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.164812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.164813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.165019 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.165245 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.165247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.165247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.165490 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.165666 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.165786 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.165887 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260928 15:02:11.166946 23712 catalog_manager.cc:1347] Generated new cluster ID: 7d6304c570b5472b9da434906ef241e9
I20260928 15:02:11.167011 23712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:11.167193 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:11.176544 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:11.168917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36743 (local address 127.1.252.254:34021)
0928 15:02:11.168965 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:11.168966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:11.168975 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:11.169023 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:11.169024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:11.169032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:11.169076 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:11.169773 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.170486 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.171655 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:11.171965 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:11.172024 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:11.173661 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:11.173663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:11.173663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:11.173670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:11.175201 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.175664 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.175665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.175666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.175719 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:11.175993 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:11.175994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:11.175994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:11.176114 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:11.176245 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:11.176385 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 15:02:11.176445 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:02:11.184218 23712 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:11.185051 23712 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:11.195907 23712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e37ed06dcf8f4d4588e942c3655bfa1c: Generated new TSK 0
I20260928 15:02:11.196128 23712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:11.201332 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23630
W20260928 15:02:11.205854 23615 connection.cc:535] client connection to 127.1.252.254:34021 recv error: Network error: failed to read from TLS socket (remote: 127.1.252.254:34021): Connection reset by peer (error 104)
2026-09-28T15:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:11 dist-test-slave-sf70 krb5kdc[23730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:11 dist-test-slave-sf70 krb5kdc[23730](info): set up 2 sockets
Sep 28 15:02:11 dist-test-slave-sf70 krb5kdc[23730](info): commencing operation
krb5kdc: starting...
W20260928 15:02:12.245777 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:12 dist-test-slave-sf70 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:12.368016 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46053
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:46053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:12.447703 23743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:12.447963 23743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:12.447988 23743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:12.449046 23743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:12.449080 23743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:12.449096 23743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:12.449111 23743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:12.449123 23743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:12.450496 23743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:46053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:12.450670 23743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:12.450832 23743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:12.455554 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:12.455592 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:12.455564 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:12.455821 23743 server_base.cc:1034] running on GCE node
I20260928 15:02:12.456240 23743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:12.456475 23743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:12.457617 23743 hybrid_clock.cc:648] HybridClock initialized: now 1790607732457605 us; error 28 us; skew 500 ppm
Sep 28 15:02:12 dist-test-slave-sf70 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607732, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:12.460043 23743 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:12.460497 23743 webserver.cc:460] Webserver started at http://127.1.252.254:37211/ using document root <none> and password file <none>
I20260928 15:02:12.460732 23743 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:12.460795 23743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:12.460925 23743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:12.461948 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "a65b9b5fcead45298a8d13ebfe164b71"
format_stamp: "Formatted at 2026-09-28 15:02:12 on dist-test-slave-sf70"
server_key: "caa2503e758cecf6e16382953bfef34f"
server_key_iv: "5ab5b14d03cdd257da10a4988cdf56ba"
server_key_version: "encryptionkey@0"
I20260928 15:02:12.462339 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "a65b9b5fcead45298a8d13ebfe164b71"
format_stamp: "Formatted at 2026-09-28 15:02:12 on dist-test-slave-sf70"
server_key: "caa2503e758cecf6e16382953bfef34f"
server_key_iv: "5ab5b14d03cdd257da10a4988cdf56ba"
server_key_version: "encryptionkey@0"
I20260928 15:02:12.463619 23743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:02:12.464469 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:12.464668 23743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:12.464753 23743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "a65b9b5fcead45298a8d13ebfe164b71"
format_stamp: "Formatted at 2026-09-28 15:02:12 on dist-test-slave-sf70"
server_key: "caa2503e758cecf6e16382953bfef34f"
server_key_iv: "5ab5b14d03cdd257da10a4988cdf56ba"
server_key_version: "encryptionkey@0"
I20260928 15:02:12.464825 23743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:12.484483 23743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:12.486464 23743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:12.486606 23743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:12.490640 23743 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:46053
I20260928 15:02:12.490674 23811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:46053 every 8 connection(s)
I20260928 15:02:12.491044 23743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:12.491945 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:12.492626 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23743
I20260928 15:02:12.492762 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:12.492966 2035 external_mini_cluster.cc:1428] Setting key e0887a145fa6c6dccb49a8bf11d4d965
I20260928 15:02:12.495199 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71: Bootstrap starting.
I20260928 15:02:12.495801 23812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:12.496150 23812 log.cc:824] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:12.496888 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71: No bootstrap required, opened a new log
I20260928 15:02:12.498678 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.493499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46237 (local address 127.1.252.254:46053)
0928 15:02:12.493695 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.493703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.493729 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:02:12.494225 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.494232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.494282 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.494438 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.494639 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.495719 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.496729 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.497205 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.497292 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.498001 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:12.498005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:12.498006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:12.498181 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:12.498212 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:12.498223 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:12.498290 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.498408 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.498453 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260928 15:02:12.499132 23812 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65b9b5fcead45298a8d13ebfe164b71" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46053 } }
I20260928 15:02:12.499302 23812 raft_consensus.cc:374] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:12.499383 23812 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65b9b5fcead45298a8d13ebfe164b71, State: Initialized, Role: FOLLOWER
I20260928 15:02:12.499524 23812 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65b9b5fcead45298a8d13ebfe164b71" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46053 } }
I20260928 15:02:12.499611 23812 raft_consensus.cc:388] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:12.499655 23812 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:12.499696 23812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:12.499809 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:12.500295 23812 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65b9b5fcead45298a8d13ebfe164b71" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46053 } }
I20260928 15:02:12.500411 23812 leader_election.cc:302] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a65b9b5fcead45298a8d13ebfe164b71; no voters:
I20260928 15:02:12.500592 23812 leader_election.cc:288] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:12.500623 23817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:12.500736 23817 raft_consensus.cc:686] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [term 1 LEADER]: Becoming Leader. State: Replica: a65b9b5fcead45298a8d13ebfe164b71, State: Running, Role: LEADER
I20260928 15:02:12.500839 23812 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:12.500877 23817 consensus_queue.cc:237] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65b9b5fcead45298a8d13ebfe164b71" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46053 } }
I20260928 15:02:12.501226 23817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65b9b5fcead45298a8d13ebfe164b71. Latest consensus state: current_term: 1 leader_uuid: "a65b9b5fcead45298a8d13ebfe164b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65b9b5fcead45298a8d13ebfe164b71" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46053 } } }
I20260928 15:02:12.501240 23818 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a65b9b5fcead45298a8d13ebfe164b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65b9b5fcead45298a8d13ebfe164b71" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 46053 } } }
I20260928 15:02:12.501391 23818 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:12.501542 23817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:12.501852 23829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:12.502557 23829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:12.504279 23829 catalog_manager.cc:1347] Generated new cluster ID: 3a3128cc63d04e2dbab816a8852cf031
I20260928 15:02:12.504339 23829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:12.505770 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:12.501291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49551 (local address 127.1.252.254:46053)
0928 15:02:12.501373 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:12.501376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:12.502525 (+ 1149us) server_negotiation.cc:407] Connection header received
0928 15:02:12.502607 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:12.502608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:12.502620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:12.502684 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:12.502859 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.503561 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.504571 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:12.504812 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:12.504904 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:12.505526 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:12.505527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:12.505528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:12.505531 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:12.505544 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:12.505548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:12.505595 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:12.505665 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 15:02:12.505673 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:02:12.540751 23829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:12.541496 23829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:12.546953 23829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a65b9b5fcead45298a8d13ebfe164b71: Generated new TSK 0
I20260928 15:02:12.547124 23829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:12.554946 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23743
W20260928 15:02:12.559381 23728 connection.cc:535] client connection to 127.1.252.254:46053 recv error: Network error: failed to read from TLS socket (remote: 127.1.252.254:46053): Connection reset by peer (error 104)
2026-09-28T15:02:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:12 dist-test-slave-sf70 krb5kdc[23843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:12 dist-test-slave-sf70 krb5kdc[23843](info): set up 2 sockets
Sep 28 15:02:12 dist-test-slave-sf70 krb5kdc[23843](info): commencing operation
krb5kdc: starting...
W20260928 15:02:13.641706 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:13 dist-test-slave-sf70 krb5kdc[23843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:13.775681 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:44253
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:13.857545 23856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:13.857755 23856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:13.857777 23856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:13.858850 23856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:13.858884 23856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:13.858898 23856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:13.858911 23856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:13.858922 23856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:13.860428 23856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:44253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:13.860620 23856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:13.860780 23856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:13.866932 23862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:13.866964 23864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:13.866933 23861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:13.866987 23856 server_base.cc:1034] running on GCE node
I20260928 15:02:13.867471 23856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:13.867762 23856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:13.868937 23856 hybrid_clock.cc:648] HybridClock initialized: now 1790607733868926 us; error 43 us; skew 500 ppm
Sep 28 15:02:13 dist-test-slave-sf70 krb5kdc[23843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607733, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:13.871461 23856 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:13.871997 23856 webserver.cc:460] Webserver started at http://127.1.252.254:45637/ using document root <none> and password file <none>
I20260928 15:02:13.872231 23856 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:13.872294 23856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:13.872429 23856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:13.873476 23856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "6d5af7c608e64dd0816018a2c07ebf9a"
format_stamp: "Formatted at 2026-09-28 15:02:13 on dist-test-slave-sf70"
server_key: "83ae45796d5cb20be55acb8904f7cafd"
server_key_iv: "5e9f761746640869b4fa765b8fbc3cfd"
server_key_version: "encryptionkey@0"
I20260928 15:02:13.873833 23856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "6d5af7c608e64dd0816018a2c07ebf9a"
format_stamp: "Formatted at 2026-09-28 15:02:13 on dist-test-slave-sf70"
server_key: "83ae45796d5cb20be55acb8904f7cafd"
server_key_iv: "5e9f761746640869b4fa765b8fbc3cfd"
server_key_version: "encryptionkey@0"
I20260928 15:02:13.875242 23856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:02:13.876164 23871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:13.876343 23856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:13.876441 23856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "6d5af7c608e64dd0816018a2c07ebf9a"
format_stamp: "Formatted at 2026-09-28 15:02:13 on dist-test-slave-sf70"
server_key: "83ae45796d5cb20be55acb8904f7cafd"
server_key_iv: "5e9f761746640869b4fa765b8fbc3cfd"
server_key_version: "encryptionkey@0"
I20260928 15:02:13.876562 23856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:13.879771 23856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:13.881776 23856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:13.881924 23856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:13.886060 23856 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:44253
I20260928 15:02:13.886111 23924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:44253 every 8 connection(s)
I20260928 15:02:13.886474 23856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:13.887497 23925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:13.889958 23925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a: Bootstrap starting.
I20260928 15:02:13.890480 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23856
I20260928 15:02:13.890610 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:13.890726 23925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:13.890846 2035 external_mini_cluster.cc:1428] Setting key a9846f5347769821cf70e1a32edde0d7
I20260928 15:02:13.891088 23925 log.cc:824] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:13.891933 23925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a: No bootstrap required, opened a new log
I20260928 15:02:13.893901 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:13.891403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56207 (local address 127.1.252.254:44253)
0928 15:02:13.891609 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:13.891618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:13.891654 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:02:13.892558 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:13.892565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:13.892593 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:13.892717 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:13.893375 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:13.893413 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:13.893414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:13.893418 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:13.893458 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:13.893470 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:13.893543 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:13.893620 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 15:02:13.893676 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260928 15:02:13.895025 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:13.895169 23925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44253 } }
I20260928 15:02:13.895328 23925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:13.895354 23925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5af7c608e64dd0816018a2c07ebf9a, State: Initialized, Role: FOLLOWER
I20260928 15:02:13.895495 23925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44253 } }
I20260928 15:02:13.895581 23925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:13.895605 23925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:13.895671 23925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:13.896371 23925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44253 } }
I20260928 15:02:13.896502 23925 leader_election.cc:302] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d5af7c608e64dd0816018a2c07ebf9a; no voters:
I20260928 15:02:13.896692 23925 leader_election.cc:288] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:13.896747 23934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:13.896989 23934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [term 1 LEADER]: Becoming Leader. State: Replica: 6d5af7c608e64dd0816018a2c07ebf9a, State: Running, Role: LEADER
I20260928 15:02:13.897032 23925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:13.897181 23934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44253 } }
I20260928 15:02:13.897797 23937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44253 } } }
I20260928 15:02:13.897800 23936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d5af7c608e64dd0816018a2c07ebf9a. Latest consensus state: current_term: 1 leader_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5af7c608e64dd0816018a2c07ebf9a" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 44253 } } }
I20260928 15:02:13.897979 23936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:13.898226 23944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:13.898336 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:13.896787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40611 (local address 127.1.252.254:44253)
0928 15:02:13.896830 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:13.896832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:13.896987 (+ 155us) server_negotiation.cc:407] Connection header received
0928 15:02:13.897072 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:13.897073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:13.897078 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:13.897116 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:13.897917 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:13.897918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:13.897919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:13.897924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:13.897940 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:13.897944 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:13.898001 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:13.898202 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 15:02:13.898215 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:02:13.898240 23937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:13.898969 23944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:13.900527 23944 catalog_manager.cc:1347] Generated new cluster ID: 00a6947182da409289d3ea3e08870e80
I20260928 15:02:13.900580 23944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:13.907757 23944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:13.907959 23944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:13.923686 23944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d5af7c608e64dd0816018a2c07ebf9a: Generated new TSK 0
I20260928 15:02:13.923941 23944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:13.925603 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23856
2026-09-28T15:02:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:13 dist-test-slave-sf70 krb5kdc[23956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:13 dist-test-slave-sf70 krb5kdc[23956](info): set up 2 sockets
Sep 28 15:02:13 dist-test-slave-sf70 krb5kdc[23956](info): commencing operation
krb5kdc: starting...
W20260928 15:02:14.966049 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:15 dist-test-slave-sf70 krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:15.090438 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33875
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:33875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:15.170349 23969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:15.170607 23969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:15.170647 23969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:15.171805 23969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:15.171873 23969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:15.171896 23969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:15.171948 23969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:15.171976 23969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:15.173471 23969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:33875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:33875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:15.173741 23969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:15.173961 23969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:15.179266 23974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:15.179262 23975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:15.179288 23977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:15.179262 23969 server_base.cc:1034] running on GCE node
I20260928 15:02:15.179796 23969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:15.180105 23969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:15.181253 23969 hybrid_clock.cc:648] HybridClock initialized: now 1790607735181240 us; error 27 us; skew 500 ppm
Sep 28 15:02:15 dist-test-slave-sf70 krb5kdc[23956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:15.183750 23969 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:15.184207 23969 webserver.cc:460] Webserver started at http://127.1.252.254:36459/ using document root <none> and password file <none>
I20260928 15:02:15.184393 23969 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:15.184479 23969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:15.184572 23969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:15.185649 23969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "de1dc4475fd0472fa445045c9ce5afda"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-sf70"
server_key: "dcea12809dd99fd520a3e71be1332772"
server_key_iv: "76a19afcd600049369ca9feb71f4af73"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.185985 23969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "de1dc4475fd0472fa445045c9ce5afda"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-sf70"
server_key: "dcea12809dd99fd520a3e71be1332772"
server_key_iv: "76a19afcd600049369ca9feb71f4af73"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.187294 23969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:15.188220 23984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:15.188418 23969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:15.188515 23969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "de1dc4475fd0472fa445045c9ce5afda"
format_stamp: "Formatted at 2026-09-28 15:02:15 on dist-test-slave-sf70"
server_key: "dcea12809dd99fd520a3e71be1332772"
server_key_iv: "76a19afcd600049369ca9feb71f4af73"
server_key_version: "encryptionkey@0"
I20260928 15:02:15.188575 23969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:15.200203 23969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:15.202224 23969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:15.202340 23969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:15.206147 23969 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:33875
I20260928 15:02:15.206220 24036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:33875 every 8 connection(s)
I20260928 15:02:15.206542 23969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:15.207425 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:15.209831 24037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda: Bootstrap starting.
I20260928 15:02:15.210449 24037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:15.210739 24037 log.cc:824] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:15.211524 24037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda: No bootstrap required, opened a new log
I20260928 15:02:15.213328 24037 raft_consensus.cc:348] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1dc4475fd0472fa445045c9ce5afda" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33875 } }
I20260928 15:02:15.213491 24037 raft_consensus.cc:374] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:15.213539 24037 raft_consensus.cc:729] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1dc4475fd0472fa445045c9ce5afda, State: Initialized, Role: FOLLOWER
I20260928 15:02:15.213703 24037 consensus_queue.cc:260] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1dc4475fd0472fa445045c9ce5afda" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33875 } }
I20260928 15:02:15.213796 24037 raft_consensus.cc:388] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:15.213845 24037 raft_consensus.cc:482] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:15.213903 24037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:15.214847 24037 raft_consensus.cc:504] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1dc4475fd0472fa445045c9ce5afda" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33875 } }
I20260928 15:02:15.214977 24037 leader_election.cc:302] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de1dc4475fd0472fa445045c9ce5afda; no voters:
I20260928 15:02:15.215167 24037 leader_election.cc:288] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:15.215193 24040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:15.215188 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 23969
I20260928 15:02:15.215337 24040 raft_consensus.cc:686] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [term 1 LEADER]: Becoming Leader. State: Replica: de1dc4475fd0472fa445045c9ce5afda, State: Running, Role: LEADER
I20260928 15:02:15.215302 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:15.215440 24037 sys_catalog.cc:564] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:15.215497 24040 consensus_queue.cc:237] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1dc4475fd0472fa445045c9ce5afda" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33875 } }
I20260928 15:02:15.215562 2035 external_mini_cluster.cc:1428] Setting key f6c038aab7f3b5ff0a89cd31cb190d58
I20260928 15:02:15.216343 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [sys.catalog]: SysCatalogTable state changed. Reason: New leader de1dc4475fd0472fa445045c9ce5afda. Latest consensus state: current_term: 1 leader_uuid: "de1dc4475fd0472fa445045c9ce5afda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1dc4475fd0472fa445045c9ce5afda" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33875 } } }
I20260928 15:02:15.216413 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de1dc4475fd0472fa445045c9ce5afda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1dc4475fd0472fa445045c9ce5afda" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 33875 } } }
I20260928 15:02:15.216538 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:15.216524 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:15.217043 24052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:15.217588 24052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:15.220293 24052 catalog_manager.cc:1347] Generated new cluster ID: 3a96f14f3eef4597be469c4b939e92b7
I20260928 15:02:15.220352 24052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:02:15 dist-test-slave-sf70 krb5kdc[23956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:15.227427 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.216122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35725 (local address 127.1.252.254:33875)
0928 15:02:15.216808 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:15.216817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:15.216852 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:02:15.216995 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:15.217003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:15.217071 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:15.217248 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:15.218007 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.219181 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.220294 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.220954 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.221067 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.223786 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:15.223792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:15.223792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:15.223808 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:15.225833 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.226210 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.226213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.226213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.226336 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.226530 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.226531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.226531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.226838 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:15.226995 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:15.227066 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 15:02:15.227206 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":240,"threads_started":1}
I20260928 15:02:15.228413 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:15.240067 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:15.232167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36069 (local address 127.1.252.254:33875)
0928 15:02:15.232208 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:15.232210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:15.232302 (+ 92us) server_negotiation.cc:407] Connection header received
0928 15:02:15.232360 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:15.232361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:15.232370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:15.232422 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:15.233036 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.233746 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.234781 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:15.235109 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:15.235149 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:15.237154 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:15.237155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:15.237156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:15.237163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:15.238679 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.239151 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.239152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.239153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.239230 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:15.239501 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:15.239502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:15.239502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:15.239646 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:15.239767 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:15.239855 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 15:02:15.239972 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:02:15.247711 24052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:15.248456 24052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:15.255813 24052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de1dc4475fd0472fa445045c9ce5afda: Generated new TSK 0
I20260928 15:02:15.256045 24052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:15.263411 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 23969
2026-09-28T15:02:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:15 dist-test-slave-sf70 krb5kdc[24068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:15 dist-test-slave-sf70 krb5kdc[24068](info): set up 2 sockets
Sep 28 15:02:15 dist-test-slave-sf70 krb5kdc[24068](info): commencing operation
krb5kdc: starting...
W20260928 15:02:16.325764 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:16 dist-test-slave-sf70 krb5kdc[24068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:16.447834 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38121
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:16.530522 24081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:16.530779 24081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:16.530822 24081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:16.532112 24081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:16.532181 24081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:16.532208 24081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:16.532243 24081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:16.532269 24081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:16.533891 24081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:16.534174 24081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:16.534415 24081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:16.539903 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:16.539909 24087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:16.540145 24081 server_base.cc:1034] running on GCE node
W20260928 15:02:16.540025 24086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:16.540512 24081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:16.540764 24081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:16.541913 24081 hybrid_clock.cc:648] HybridClock initialized: now 1790607736541900 us; error 30 us; skew 500 ppm
Sep 28 15:02:16 dist-test-slave-sf70 krb5kdc[24068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607736, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:16.552181 24081 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:16.552690 24081 webserver.cc:460] Webserver started at http://127.1.252.254:37123/ using document root <none> and password file <none>
I20260928 15:02:16.552938 24081 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:16.553005 24081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:16.553148 24081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:16.554216 24081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2"
format_stamp: "Formatted at 2026-09-28 15:02:16 on dist-test-slave-sf70"
server_key: "bb8e8abfa926573d94bf9a7348171f18"
server_key_iv: "5d899cc4ba65c49efe4eee0421745d90"
server_key_version: "encryptionkey@0"
I20260928 15:02:16.554584 24081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2"
format_stamp: "Formatted at 2026-09-28 15:02:16 on dist-test-slave-sf70"
server_key: "bb8e8abfa926573d94bf9a7348171f18"
server_key_iv: "5d899cc4ba65c49efe4eee0421745d90"
server_key_version: "encryptionkey@0"
I20260928 15:02:16.556006 24081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:16.556793 24096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:16.556977 24081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:16.557080 24081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2"
format_stamp: "Formatted at 2026-09-28 15:02:16 on dist-test-slave-sf70"
server_key: "bb8e8abfa926573d94bf9a7348171f18"
server_key_iv: "5d899cc4ba65c49efe4eee0421745d90"
server_key_version: "encryptionkey@0"
I20260928 15:02:16.557157 24081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:16.571478 24081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:16.573678 24081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:16.573853 24081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:16.578281 24081 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:38121
I20260928 15:02:16.578313 24149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:38121 every 8 connection(s)
I20260928 15:02:16.578693 24081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:16.579579 24150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:16.581883 24150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2: Bootstrap starting.
I20260928 15:02:16.582479 24150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:16.582767 24150 log.cc:824] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:16.582755 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24081
I20260928 15:02:16.582911 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:16.583096 2035 external_mini_cluster.cc:1428] Setting key 91a4a095830c7d17be95b059623d3532
I20260928 15:02:16.583529 24150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2: No bootstrap required, opened a new log
I20260928 15:02:16.585376 24150 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38121 } }
I20260928 15:02:16.585631 24150 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:16.585654 24150 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9e2ea4d82c4e14925a98db6c67e3f2, State: Initialized, Role: FOLLOWER
I20260928 15:02:16.585830 24150 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38121 } }
I20260928 15:02:16.585925 24150 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:16.585970 24150 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:16.586018 24150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:16.586596 24150 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38121 } }
I20260928 15:02:16.586710 24150 leader_election.cc:302] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee9e2ea4d82c4e14925a98db6c67e3f2; no voters:
I20260928 15:02:16.586877 24150 leader_election.cc:288] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:16.586931 24155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:16.587072 24155 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [term 1 LEADER]: Becoming Leader. State: Replica: ee9e2ea4d82c4e14925a98db6c67e3f2, State: Running, Role: LEADER
I20260928 15:02:16.587132 24150 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:16.587208 24155 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38121 } }
I20260928 15:02:16.587561 24155 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9e2ea4d82c4e14925a98db6c67e3f2. Latest consensus state: current_term: 1 leader_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38121 } } }
I20260928 15:02:16.587574 24156 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9e2ea4d82c4e14925a98db6c67e3f2" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38121 } } }
I20260928 15:02:16.587656 24155 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:16.587666 24156 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:16.588099 24162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:16.588919 24162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:16.590885 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.583636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49519 (local address 127.1.252.254:38121)
0928 15:02:16.583836 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.583846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.583882 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:02:16.585363 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.585372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.585426 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.585689 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.585927 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.587520 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.588752 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.589295 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.589414 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.590101 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.590104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.590105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:16.590323 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.590367 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:16.590384 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.590449 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.590590 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.590658 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260928 15:02:16.591037 24162 catalog_manager.cc:1347] Generated new cluster ID: 33bedd65b0e544ea83417a25f2120385
I20260928 15:02:16.591078 24162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:16.591995 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:16.596892 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:16.593500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41417 (local address 127.1.252.254:38121)
0928 15:02:16.593534 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:16.593536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:16.593607 (+ 71us) server_negotiation.cc:407] Connection header received
0928 15:02:16.593659 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:16.593660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:16.593669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:16.593737 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:16.593865 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.594582 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.595554 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:16.595934 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:16.596008 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:16.596645 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:16.596646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:16.596647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:16.596651 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:16.596667 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:16.596672 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:16.596730 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:16.596784 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 15:02:16.596814 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:16.603868 24162 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:16.604445 24162 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:16.609428 24162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee9e2ea4d82c4e14925a98db6c67e3f2: Generated new TSK 0
I20260928 15:02:16.609597 24162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:16.615393 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24081
2026-09-28T15:02:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:16 dist-test-slave-sf70 krb5kdc[24181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:16 dist-test-slave-sf70 krb5kdc[24181](info): set up 2 sockets
Sep 28 15:02:16 dist-test-slave-sf70 krb5kdc[24181](info): commencing operation
krb5kdc: starting...
W20260928 15:02:17.673862 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:17 dist-test-slave-sf70 krb5kdc[24181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:17.803820 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:39729
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:17.887545 24194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:17.887807 24194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:17.887849 24194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:17.889043 24194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:17.889115 24194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:17.889144 24194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:17.889176 24194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:17.889205 24194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:17.890726 24194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:39729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:17.891028 24194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:17.891287 24194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:17.896302 24199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:17.896302 24200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:17.896521 24194 server_base.cc:1034] running on GCE node
W20260928 15:02:17.896302 24202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:17.896883 24194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:17.897188 24194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:17.898425 24194 hybrid_clock.cc:648] HybridClock initialized: now 1790607737898389 us; error 52 us; skew 500 ppm
Sep 28 15:02:17 dist-test-slave-sf70 krb5kdc[24181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:17.901086 24194 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:17.901716 24194 webserver.cc:460] Webserver started at http://127.1.252.254:40853/ using document root <none> and password file <none>
I20260928 15:02:17.902042 24194 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:17.902122 24194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:17.902272 24194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:17.903357 24194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "ce290c24e3f74ab986f11761794c269b"
format_stamp: "Formatted at 2026-09-28 15:02:17 on dist-test-slave-sf70"
server_key: "68111123a46d4fb8a2a740b404af9b8e"
server_key_iv: "f1593e1bde3c050e7662f496c0a43ae5"
server_key_version: "encryptionkey@0"
I20260928 15:02:17.903734 24194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "ce290c24e3f74ab986f11761794c269b"
format_stamp: "Formatted at 2026-09-28 15:02:17 on dist-test-slave-sf70"
server_key: "68111123a46d4fb8a2a740b404af9b8e"
server_key_iv: "f1593e1bde3c050e7662f496c0a43ae5"
server_key_version: "encryptionkey@0"
I20260928 15:02:17.905269 24194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:17.906213 24209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:17.906442 24194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:17.906566 24194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "ce290c24e3f74ab986f11761794c269b"
format_stamp: "Formatted at 2026-09-28 15:02:17 on dist-test-slave-sf70"
server_key: "68111123a46d4fb8a2a740b404af9b8e"
server_key_iv: "f1593e1bde3c050e7662f496c0a43ae5"
server_key_version: "encryptionkey@0"
I20260928 15:02:17.906647 24194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:17.910320 24194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:17.912642 24194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:17.912796 24194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:17.917063 24194 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:39729
I20260928 15:02:17.917100 24262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:39729 every 8 connection(s)
I20260928 15:02:17.917438 24194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:17.918535 24263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:17.918594 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24194
I20260928 15:02:17.918707 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:17.918915 2035 external_mini_cluster.cc:1428] Setting key 423b3b098e476592888d6a9e2e85b1a4
I20260928 15:02:17.921547 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.919572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54381 (local address 127.1.252.254:39729)
0928 15:02:17.919749 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.919757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.919781 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:17.920274 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.920282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.920315 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.920430 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.921143 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:17.921147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:17.921147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:17.921151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:17.921198 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:17.921212 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:17.921292 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.921325 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.921363 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 15:02:17.922365 24263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b: Bootstrap starting.
I20260928 15:02:17.922753 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:17.923183 24263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:17.923533 24263 log.cc:824] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:17.924443 24263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b: No bootstrap required, opened a new log
I20260928 15:02:17.925748 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:17.924536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55681 (local address 127.1.252.254:39729)
0928 15:02:17.924571 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:17.924573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:17.924770 (+ 197us) server_negotiation.cc:407] Connection header received
0928 15:02:17.924843 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:17.924845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:17.924851 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:17.924911 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:17.925541 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:17.925543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:17.925543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:17.925547 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:17.925564 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:17.925570 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:17.925638 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:17.925674 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 15:02:17.925688 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 15:02:17.927039 24263 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce290c24e3f74ab986f11761794c269b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39729 } }
I20260928 15:02:17.927205 24263 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:17.927251 24263 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce290c24e3f74ab986f11761794c269b, State: Initialized, Role: FOLLOWER
I20260928 15:02:17.927384 24263 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce290c24e3f74ab986f11761794c269b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39729 } }
I20260928 15:02:17.927496 24263 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:17.927533 24263 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:17.927575 24263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:17.928272 24263 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce290c24e3f74ab986f11761794c269b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39729 } }
I20260928 15:02:17.928400 24263 leader_election.cc:302] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce290c24e3f74ab986f11761794c269b; no voters:
I20260928 15:02:17.928574 24263 leader_election.cc:288] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:17.928674 24273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:17.928828 24263 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:17.928848 24273 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [term 1 LEADER]: Becoming Leader. State: Replica: ce290c24e3f74ab986f11761794c269b, State: Running, Role: LEADER
I20260928 15:02:17.928928 24273 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce290c24e3f74ab986f11761794c269b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39729 } }
I20260928 15:02:17.929549 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce290c24e3f74ab986f11761794c269b. Latest consensus state: current_term: 1 leader_uuid: "ce290c24e3f74ab986f11761794c269b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce290c24e3f74ab986f11761794c269b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39729 } } }
I20260928 15:02:17.929616 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:17.929459 24274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce290c24e3f74ab986f11761794c269b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce290c24e3f74ab986f11761794c269b" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 39729 } } }
I20260928 15:02:17.929697 24274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:17.930289 24281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:17.930764 24281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:17.932333 24281 catalog_manager.cc:1347] Generated new cluster ID: 3d8a839f869842aea014b1182ca57e5e
I20260928 15:02:17.932386 24281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:17.961649 24281 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:17.961851 24281 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:17.966140 24281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce290c24e3f74ab986f11761794c269b: Generated new TSK 0
I20260928 15:02:17.966320 24281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:17.971158 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24194
2026-09-28T15:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1355 ms)
[----------] 18 tests from AuthTokenIssuingTest (24484 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:18 dist-test-slave-sf70 krb5kdc[24294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:18 dist-test-slave-sf70 krb5kdc[24294](info): set up 2 sockets
Sep 28 15:02:18 dist-test-slave-sf70 krb5kdc[24294](info): commencing operation
krb5kdc: starting...
W20260928 15:02:19.025893 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:19.148131 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38827
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:38827,127.1.252.253:36045,127.1.252.252:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:19.233147 24307 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:19.233367 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:19.233387 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:19.233592 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:19.234754 24307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:19.234794 24307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:19.234808 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:19.234820 24307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:19.234833 24307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:19.236310 24307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:38827,127.1.252.253:36045,127.1.252.252:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:19.236509 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:19.236681 24307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:19.241647 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:19.241698 24307 server_base.cc:1034] running on GCE node
W20260928 15:02:19.241658 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:19.241668 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:19.242106 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:19.242403 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:19.243558 24307 hybrid_clock.cc:648] HybridClock initialized: now 1790607739243530 us; error 60 us; skew 500 ppm
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:19.246207 24307 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:19.246588 24307 webserver.cc:460] Webserver started at http://127.1.252.254:33933/ using document root <none> and password file <none>
I20260928 15:02:19.246832 24307 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:19.246912 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:19.247078 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:19.248238 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "b34ec3c34b2e4b9592addbe3f10d6243"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "8bff005e171e7b0fd220aa239a5036f9"
server_key_iv: "0f72e0568eca0a709a8046a96b3115fd"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.248598 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "b34ec3c34b2e4b9592addbe3f10d6243"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "8bff005e171e7b0fd220aa239a5036f9"
server_key_iv: "0f72e0568eca0a709a8046a96b3115fd"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.249975 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:19.250777 24322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:19.250993 24307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:19.251109 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "b34ec3c34b2e4b9592addbe3f10d6243"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "8bff005e171e7b0fd220aa239a5036f9"
server_key_iv: "0f72e0568eca0a709a8046a96b3115fd"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.251293 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:19.278498 24307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:19.280982 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:19.281152 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:19.285301 24307 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:38827
I20260928 15:02:19.285334 24375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:38827 every 8 connection(s)
I20260928 15:02:19.285840 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:19.286898 24376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:19.288611 24376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:19.293581 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24307
I20260928 15:02:19.293711 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:19.293922 2035 external_mini_cluster.cc:1428] Setting key a1d52a743d345125f80a8009b07a1cd3
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:19.298821 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.289345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.1.252.254:38827)
0928 15:02:19.289532 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.289541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.289627 (+ 86us) server_negotiation.cc:407] Connection header received
0928 15:02:19.290106 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.290107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.290124 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.290207 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.290721 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.291644 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.292571 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.292876 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.292970 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.295323 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.295331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.295331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.295340 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.297028 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.297549 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.297552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.297553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.297639 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.297978 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.297979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.297979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.298198 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.298276 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.298549 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.298655 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260928 15:02:19.298954 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.289324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38827 (local address 127.0.0.1:47820)
0928 15:02:19.289499 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.289537 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.289983 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.290212 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.290214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.290453 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.290605 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.290608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.291763 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.291764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.292486 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.292488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.292980 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.292981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.293036 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.293579 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.293591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.295216 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.297128 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.297135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.297138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.297439 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.297714 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.297715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.297716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.297832 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.298287 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.298295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.298382 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.298506 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.298767 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":76,"thread_start_us":97,"threads_started":1}
I20260928 15:02:19.299815 24376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:19.300241 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.300105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52792)
0928 15:02:19.300161 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.300211 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:02:19.300443 24324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
I20260928 15:02:19.300940 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.300824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52796)
0928 15:02:19.300877 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.300908 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 15:02:19.301059 24376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:36045: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.252.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab.
Entry for principal kudu/127.1.252.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab.
I20260928 15:02:19.319705 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:36045
--webserver_interface=127.1.252.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab
--principal=kudu/127.1.252.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:38827,127.1.252.253:36045,127.1.252.252:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:19.350310 24376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } attempt: 1
I20260928 15:02:19.350939 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.350781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52806)
0928 15:02:19.350854 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.350903 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:02:19.351362 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.351271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52812)
0928 15:02:19.351313 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.351342 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 15:02:19.351500 24376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:36045: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
W20260928 15:02:19.404528 24383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:19.404769 24383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:19.404789 24383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:19.404814 24383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:19.406122 24383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:19.406168 24383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:19.406184 24383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:19.406198 24383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:19.406211 24383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:19.407862 24383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:38827,127.1.252.253:36045,127.1.252.252:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab
--principal=kudu/127.1.252.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.252.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:19.408167 24383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:19.408459 24383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:19.413827 24388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:19.413836 24391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:19.413836 24389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:19.414007 24383 server_base.cc:1034] running on GCE node
I20260928 15:02:19.414260 24383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:19.414580 24383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:19.415762 24383 hybrid_clock.cc:648] HybridClock initialized: now 1790607739415745 us; error 50 us; skew 500 ppm
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:19.418406 24383 init.cc:377] Logged in from keytab as kudu/127.1.252.253@KRBTEST.COM (short username kudu)
I20260928 15:02:19.418864 24383 webserver.cc:460] Webserver started at http://127.1.252.253:41433/ using document root <none> and password file <none>
I20260928 15:02:19.419054 24383 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:19.419096 24383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:19.419188 24383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:19.419704 24376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } attempt: 2
I20260928 15:02:19.420182 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.420034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52814)
0928 15:02:19.420114 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.420151 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:02:19.420434 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data/instance:
uuid: "5742a0d3900943468175847d33f2ca83"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "c6a590e15e9cc04a429de5a4f1faef2b"
server_key_iv: "1b04e06cdeabe88f168d00d1474c3ae4"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.420626 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.420517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52820)
0928 15:02:19.420570 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.420601 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:02:19.420728 24376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:36045: Network error: Client connection negotiation failed: client connection to 127.1.252.253:36045: connect: Connection refused (error 111)
I20260928 15:02:19.420861 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal/instance:
uuid: "5742a0d3900943468175847d33f2ca83"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "c6a590e15e9cc04a429de5a4f1faef2b"
server_key_iv: "1b04e06cdeabe88f168d00d1474c3ae4"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.422287 24383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:19.423158 24398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:19.423349 24383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:19.423446 24383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal
uuid: "5742a0d3900943468175847d33f2ca83"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "c6a590e15e9cc04a429de5a4f1faef2b"
server_key_iv: "1b04e06cdeabe88f168d00d1474c3ae4"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.423666 24383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:19.435629 24383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:19.438005 24383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:19.438215 24383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:19.442446 24383 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.253:36045
I20260928 15:02:19.442489 24450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.253:36045 every 8 connection(s)
I20260928 15:02:19.442864 24383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
I20260928 15:02:19.444002 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:19.444928 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24383
I20260928 15:02:19.445076 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-1/wal/instance
I20260928 15:02:19.445284 2035 external_mini_cluster.cc:1428] Setting key ec8fbacb74b6ea6068b7cf8edbd0c501
I20260928 15:02:19.445730 24451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.252.252@KRBTEST.COM; defaulting to no policy
I20260928 15:02:19.457027 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.446495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.1.252.254:38827)
0928 15:02:19.446559 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.446563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.446811 (+ 248us) server_negotiation.cc:407] Connection header received
0928 15:02:19.447848 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.447850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.447864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.447962 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.448605 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.449454 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.450460 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.450691 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.450783 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.453688 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.453690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.453690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.453698 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.455481 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.455980 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.455982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.455982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.456047 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.456346 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.456347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.456347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.456473 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.456552 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.456837 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.456924 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.252@KRBTEST.COM" created.
I20260928 15:02:19.457243 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.446437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38827 (local address 127.0.0.1:47824)
0928 15:02:19.446649 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.446700 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.447667 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.448042 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.448046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.448295 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.448494 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.448499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.449557 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.449558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.450358 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.450360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.450805 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.450807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.450946 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.451562 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.451570 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.453529 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.455589 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.455598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.455601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.455849 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.456119 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.456121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.456122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.456247 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.456544 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.456551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.456702 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.456816 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.456992 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260928 15:02:19.457995 24451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:19.465803 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.458278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52824)
0928 15:02:19.458474 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.458484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.458522 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.458667 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.458668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.458783 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.458836 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.458837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.459796 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.459797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.460381 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.460383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.460864 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.460865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.460953 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.461390 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.461396 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.462845 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.464681 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.464682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.464682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.464849 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.465074 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.465075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.465075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.465127 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.465435 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.465436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.465479 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.465559 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.465668 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 15:02:19.465879 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.458331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.1.252.253:36045)
0928 15:02:19.458501 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.458505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.458531 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:02:19.458557 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.458559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.458574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.458666 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.458912 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.459649 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.460472 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.460776 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.460865 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.462965 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.462974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.462974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.462981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.464578 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.464947 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.464950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.464951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.465007 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.465199 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.465200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.465200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.465344 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.465444 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.465605 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.465744 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 15:02:19.466625 24451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:19.467043 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.466897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51114)
0928 15:02:19.466956 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.467008 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 15:02:19.467229 24402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
I20260928 15:02:19.467796 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.467681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51124)
0928 15:02:19.467740 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.467766 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 15:02:19.467921 24451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:37805: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab.
Entry for principal kudu/127.1.252.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab.
I20260928 15:02:19.477344 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:37805
--webserver_interface=127.1.252.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab
--principal=kudu/127.1.252.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:38827,127.1.252.253:36045,127.1.252.252:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:19.511957 24376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } attempt: 3
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:19.517135 24451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } attempt: 1
I20260928 15:02:19.517589 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.517395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51128)
0928 15:02:19.517465 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.517544 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 15:02:19.518246 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.518138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51138)
0928 15:02:19.518188 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.518218 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:02:19.518328 24451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:37805: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
I20260928 15:02:19.519665 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.512549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.1.252.253:36045)
0928 15:02:19.512619 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.512622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.512634 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:19.512664 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.512665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.512677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.512728 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.513060 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.513629 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.514338 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.514574 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.514662 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.516820 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.516821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.516821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.516829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.518580 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.518948 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.518949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.518949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.519020 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.519212 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.519213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.519213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.519327 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.519380 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.519551 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.519614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:02:19.519704 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.512453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52834)
0928 15:02:19.512524 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.512535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.512596 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.512732 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.512733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.512896 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.512958 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.512959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.513721 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.513723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.514236 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.514238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.514663 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.514664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.514750 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.515248 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.515252 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.516726 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.518656 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.518657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.518657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.518870 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.519087 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.519087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.519088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.519152 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.519393 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.519394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.519445 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.519517 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.519611 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 15:02:19.520148 24376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:19.520493 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.520383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51152)
0928 15:02:19.520433 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.520468 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 15:02:19.520843 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.520754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51168)
0928 15:02:19.520799 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.520825 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 15:02:19.520939 24376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:37805: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
I20260928 15:02:19.552083 24376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } attempt: 1
I20260928 15:02:19.552625 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.552476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51172)
0928 15:02:19.552544 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.552590 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 15:02:19.553025 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.552923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51178)
0928 15:02:19.552973 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.553001 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 15:02:19.553161 24376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:37805: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
W20260928 15:02:19.562407 24458 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:19.562646 24458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:19.562685 24458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:19.562724 24458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:19.563850 24458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:19.563931 24458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:19.563961 24458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:19.563988 24458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:19.564010 24458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:19.565479 24458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:38827,127.1.252.253:36045,127.1.252.252:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab
--principal=kudu/127.1.252.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.252.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:19.565759 24458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:19.565994 24458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:19.570900 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:19.570900 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:19.571116 24458 server_base.cc:1034] running on GCE node
W20260928 15:02:19.570914 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:19.571511 24458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:19.571820 24458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:19.573057 24458 hybrid_clock.cc:648] HybridClock initialized: now 1790607739573033 us; error 32 us; skew 500 ppm
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:19.575573 24458 init.cc:377] Logged in from keytab as kudu/127.1.252.252@KRBTEST.COM (short username kudu)
I20260928 15:02:19.576074 24458 webserver.cc:460] Webserver started at http://127.1.252.252:41921/ using document root <none> and password file <none>
I20260928 15:02:19.576304 24458 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:19.576372 24458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:19.576516 24458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:19.577592 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data/instance:
uuid: "9ee0c011d7d94b7797586c886907a057"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "70fff92b7f84d7ea47f852cf7a349158"
server_key_iv: "e46537e06d194e35cd50ce2052d23e81"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.577957 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal/instance:
uuid: "9ee0c011d7d94b7797586c886907a057"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "70fff92b7f84d7ea47f852cf7a349158"
server_key_iv: "e46537e06d194e35cd50ce2052d23e81"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.579264 24458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:02:19.580190 24473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:19.580463 24458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:19.580577 24458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal
uuid: "9ee0c011d7d94b7797586c886907a057"
format_stamp: "Formatted at 2026-09-28 15:02:19 on dist-test-slave-sf70"
server_key: "70fff92b7f84d7ea47f852cf7a349158"
server_key_iv: "e46537e06d194e35cd50ce2052d23e81"
server_key_version: "encryptionkey@0"
I20260928 15:02:19.580667 24458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:19.586462 24451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } attempt: 2
I20260928 15:02:19.586891 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.586762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51190)
0928 15:02:19.586818 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.586858 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 15:02:19.587251 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.587171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51200)
0928 15:02:19.587210 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.587234 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 15:02:19.587342 24451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:37805: Network error: Client connection negotiation failed: client connection to 127.1.252.252:37805: connect: Connection refused (error 111)
I20260928 15:02:19.597146 24458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:19.599228 24458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:19.599392 24458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:19.603363 24458 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.252:37805
I20260928 15:02:19.603459 24526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.252:37805 every 8 connection(s)
I20260928 15:02:19.603806 24458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
I20260928 15:02:19.604957 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:19.606523 24527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:19.613752 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24458
I20260928 15:02:19.613883 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607474232031-2035-0/minicluster-data/master-2/wal/instance
I20260928 15:02:19.614099 2035 external_mini_cluster.cc:1428] Setting key 5ad5d30155aefdc06dd278e5501ebb72
I20260928 15:02:19.616055 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.607223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.1.252.254:38827)
0928 15:02:19.607381 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.607383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.607550 (+ 167us) server_negotiation.cc:407] Connection header received
0928 15:02:19.608073 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.608074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.608084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.608145 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.608800 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.609281 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.610406 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.610604 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.610662 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.612794 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.612795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.612796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.612803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.614378 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.614944 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.614945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.614946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.615015 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.615315 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.615315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.615316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.615431 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.615481 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.615842 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.615956 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:02:19.616199 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.607168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:38827 (local address 127.0.0.1:47828)
0928 15:02:19.607365 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.607406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.607970 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.608222 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.608225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.608506 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.608715 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.608719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.609366 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.609367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.610326 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.610329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.610692 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.610692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.610795 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.611254 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.611262 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.612696 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.614491 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.614505 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.614506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.614868 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.615088 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.615089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.615089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.615244 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.615500 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.615507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.615676 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.615770 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.615934 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260928 15:02:19.616771 24527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:19.624383 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.617140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.1.252.253:36045)
0928 15:02:19.617183 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.617185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.617194 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:19.617254 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.617255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.617263 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.617307 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.617621 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.618418 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.619404 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.619608 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.619662 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.621432 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.621433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.621434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.621441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.622920 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.623297 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.623298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.623299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.623365 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.623720 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.623721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.623721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.623877 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.624002 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.624202 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.624283 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:02:19.624462 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.614479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.1.252.252:37805)
0928 15:02:19.614659 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.614663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.614721 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:02:19.614793 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.614794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.614804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.614867 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.615666 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.616718 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.617749 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.618249 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.618341 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.621021 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.621029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.621030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.621036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.622853 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.623306 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.623309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.623309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.623365 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.623720 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.623721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.623721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.623932 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.624091 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.624202 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.624320 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 15:02:19.624418 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.617081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:36045 (local address 127.0.0.1:52836)
0928 15:02:19.617135 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.617146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.617188 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.617319 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.617320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.617464 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.617514 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.617515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.618540 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.618541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.619308 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.619312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.619683 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.619684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.619765 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.620215 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.620219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.621330 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.622994 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.622995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.622995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.623210 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.623572 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.623573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.623573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.623642 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.624033 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.624034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.624077 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.624156 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.624264 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 15:02:19.625365 24527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:19.628315 24376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } attempt: 2
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:19.633484 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.625627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51228)
0928 15:02:19.626048 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.626059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.626104 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.626239 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.626240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.626374 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.626408 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.626409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.627077 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.627078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.627616 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.627619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.628097 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.628098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.628148 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.628475 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.628481 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.630487 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.632429 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.632430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.632430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.632590 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.632785 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.632786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.632787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.632866 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.633193 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.633194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.633234 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.633293 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.633372 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405}
I20260928 15:02:19.633780 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.625307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.1.252.254:38827)
0928 15:02:19.625357 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.625359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.625367 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:19.625389 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.625390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.625399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.625471 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.626039 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.626709 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.627442 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.627706 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.627779 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.630653 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.630654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.630655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.630661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.632666 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.633015 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.633017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.633017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.633078 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.633280 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.633281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.633282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.633398 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.633571 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.633607 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.633676 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:02:19.633800 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.625650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.1.252.252:37805)
0928 15:02:19.625684 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.625686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.626144 (+ 458us) server_negotiation.cc:407] Connection header received
0928 15:02:19.626166 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.626166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.626175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.626216 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.626496 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.626997 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.627706 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.628015 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.628097 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.630607 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.630609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.630609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.630616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.632330 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.632681 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.632682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.632682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.632730 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.632942 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.632943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.632943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.633106 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.633170 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.633649 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.633723 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:02:19.635665 24527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: Bootstrap starting.
I20260928 15:02:19.636309 24527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:19.636538 24527 log.cc:824] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:19.637022 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.628597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51232)
0928 15:02:19.628664 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.628676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.628728 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.629750 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.629751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.629896 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.629927 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.629929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.630824 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.630825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.631621 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.631624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.632640 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.632641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.632704 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.633084 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.633089 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.634438 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.635963 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.635964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.635964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.636134 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.636346 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.636347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.636348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.636404 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.636678 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.636679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.636722 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.636809 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.636899 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 15:02:19.637368 24527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: No bootstrap required, opened a new log
I20260928 15:02:19.637485 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.628627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.1.252.252:37805)
0928 15:02:19.629597 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.629599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.629607 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:19.629627 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.629627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.629637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.629737 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.630014 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.630744 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.631699 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.632553 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.632632 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.634539 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.634541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.634541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.634547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.635843 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.636210 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.636211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.636211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.636268 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.636476 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.636477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.636477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.636608 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.636684 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.637222 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.637313 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":95,"threads_started":1}
I20260928 15:02:19.638682 24376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243: Bootstrap starting.
I20260928 15:02:19.639343 24527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.639505 24527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee0c011d7d94b7797586c886907a057, State: Initialized, Role: FOLLOWER
I20260928 15:02:19.639597 24376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:19.639593 24527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.639878 24527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:19.639952 24376 log.cc:824] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:19.640807 24376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243: No bootstrap required, opened a new log
I20260928 15:02:19.641367 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.641475 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
W20260928 15:02:19.642028 24546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:19.642122 24546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:19.643083 24376 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.643218 24376 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34ec3c34b2e4b9592addbe3f10d6243, State: Initialized, Role: FOLLOWER
I20260928 15:02:19.643293 24376 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.643603 24376 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:19.643571 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.643754 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:19.645464 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.634397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.1.252.253:36045)
0928 15:02:19.636820 (+ 2423us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.636823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.636833 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:19.636859 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.636860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.636872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.636947 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.637544 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.638246 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.639212 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.639487 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.639562 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.642411 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.642413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.642414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.642422 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.644104 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.644476 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.644477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.644477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.644534 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.644755 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.644756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.644756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.644890 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.645031 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.645263 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.645346 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2397}
W20260928 15:02:19.646167 24559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:19.646215 24559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:19.647464 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:19.648465 24349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b34ec3c34b2e4b9592addbe3f10d6243"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832
I20260928 15:02:19.648588 24349 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:02:19.648638 24349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:19.647070 24458 master_runner.cc:186] Error getting master registration for 127.1.252.253:36045: OK, instance_id { permanent_uuid: "5742a0d3900943468175847d33f2ca83" instance_seqno: 1790607739441283 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:19.649403 24458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:19.649546 24349 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.649756 24349 leader_election.cc:288] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [CANDIDATE]: Term 1 election: Requested vote from peers 5742a0d3900943468175847d33f2ca83 (127.1.252.253:36045), 9ee0c011d7d94b7797586c886907a057 (127.1.252.252:37805)
I20260928 15:02:19.647046 24307 master_runner.cc:186] Error getting master registration for 127.1.252.253:36045: OK, instance_id { permanent_uuid: "5742a0d3900943468175847d33f2ca83" instance_seqno: 1790607739441283 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:19.650063 24307 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:19.650028 24501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ee0c011d7d94b7797586c886907a057"
I20260928 15:02:19.650151 24501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:19.651157 24501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b34ec3c34b2e4b9592addbe3f10d6243 in term 1.
I20260928 15:02:19.651371 24326 leader_election.cc:302] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee0c011d7d94b7797586c886907a057, b34ec3c34b2e4b9592addbe3f10d6243; no voters:
I20260928 15:02:19.651569 24548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:19.651814 24548 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [term 1 LEADER]: Becoming Leader. State: Replica: b34ec3c34b2e4b9592addbe3f10d6243, State: Running, Role: LEADER
I20260928 15:02:19.651979 24548 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.652554 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34ec3c34b2e4b9592addbe3f10d6243. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.652719 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:19.663471 24568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:19.663725 24568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:19.667346 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.653296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.1.252.253:36045)
0928 15:02:19.653708 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.653710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.655303 (+ 1593us) server_negotiation.cc:407] Connection header received
0928 15:02:19.656261 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.656262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.656276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.656336 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.657465 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.658181 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.659616 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.659964 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.660010 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.662212 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.662226 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.662227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.662234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.665570 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.666155 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.666156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.666156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.666228 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.666534 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.666535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.666536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.666695 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.666786 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.667149 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.667233 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260928 15:02:19.668658 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.653107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.1.252.252:37805)
0928 15:02:19.653148 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.653150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.653837 (+ 687us) server_negotiation.cc:407] Connection header received
0928 15:02:19.653906 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.653907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.653925 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.653975 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.655236 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.656100 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.658286 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.658612 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.658657 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.662217 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.662219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.662219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.662227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.666347 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.667024 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.667025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.667026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.667088 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.667419 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.667421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.667421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.667570 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.667660 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.668469 (+ 809us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.668563 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:02:19.669183 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.652659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.1.252.254:38827)
0928 15:02:19.652707 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.652709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.653501 (+ 792us) server_negotiation.cc:407] Connection header received
0928 15:02:19.653620 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.653621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.653633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.653665 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.655424 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.656169 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.658738 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.659046 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.659087 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.662527 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.662529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.662529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.662536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.665563 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.667296 (+ 1733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.667297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.667298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.667348 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.668019 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.668020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.668020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.668167 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.668236 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.669015 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.669100 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:02:19.676862 24501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [term 1 FOLLOWER]: Refusing update from remote peer b34ec3c34b2e4b9592addbe3f10d6243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:19.677209 24561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:19.678452 24451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } attempt: 3
I20260928 15:02:19.679054 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34ec3c34b2e4b9592addbe3f10d6243. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.679148 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:19.680382 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.680483 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:19.680570 24571 mvcc.cc:205] Tried to move back new op lower bound from 7334329301713584128 to 7334329301615747072. Current Snapshot: MvccSnapshot[applied={T|T < 7334329301713584128}]
I20260928 15:02:19.681042 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.681114 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:19.681344 24572 mvcc.cc:205] Tried to move back new op lower bound from 7334329301713584128 to 7334329301615747072. Current Snapshot: MvccSnapshot[applied={T|T < 7334329301713584128}]
Sep 28 15:02:19 dist-test-slave-sf70 krb5kdc[24294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:02:19.683604 24568 catalog_manager.cc:1347] Generated new cluster ID: 82445cffbeb94f2d89be859d0eda3eb5
I20260928 15:02:19.683651 24568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:19.687351 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.678730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:37805 (local address 127.0.0.1:51244)
0928 15:02:19.678813 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:19.678824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:19.678868 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:19.679067 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:19.679068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:19.679259 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:19.679310 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.679313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.680253 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.680254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.681070 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:19.681073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.681667 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.681668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:19.681728 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.682220 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:19.682225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:19.683777 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:19.686203 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.686205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.686205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.686429 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.686682 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:19.686683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:19.686684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:19.686753 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:19.687014 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:19.687015 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:19.687078 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 15:02:19.687141 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 15:02:19.687251 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:02:19.687487 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:19.678776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.1.252.252:37805)
0928 15:02:19.678944 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.678946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.678952 (+ 6us) server_negotiation.cc:407] Connection header received
0928 15:02:19.678988 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.678989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.678998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.679045 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.679390 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.680154 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.681204 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:19.681550 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:19.681614 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:19.683868 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:19.683869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:19.683870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:19.683877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:19.686123 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.686522 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.686523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.686524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.686600 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:19.686819 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:19.686820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:19.686821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:19.686945 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:19.687015 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:19.687165 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:02:19.687310 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 15:02:19.689287 24451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: Bootstrap starting.
I20260928 15:02:19.690027 24451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:19.690328 24451 log.cc:824] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:19.691159 24451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: No bootstrap required, opened a new log
I20260928 15:02:19.693310 24451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.693434 24451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a0d3900943468175847d33f2ca83, State: Initialized, Role: FOLLOWER
I20260928 15:02:19.693512 24451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } }
I20260928 15:02:19.693799 24451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:19.693881 24424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:19.694903 24424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [term 1 FOLLOWER]: Refusing update from remote peer b34ec3c34b2e4b9592addbe3f10d6243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:19.695114 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5742a0d3900943468175847d33f2ca83"
I20260928 15:02:19.695845 24425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b34ec3c34b2e4b9592addbe3f10d6243 in term 1.
I20260928 15:02:19.696188 24548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:02:19.698959 24588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:19.699005 24588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:19.699179 24577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.699234 24577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:19.699270 24577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.699301 24577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:19.699481 24589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34ec3c34b2e4b9592addbe3f10d6243. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.699539 24589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:19.705458 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34ec3c34b2e4b9592addbe3f10d6243" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 38827 } } peers { permanent_uuid: "5742a0d3900943468175847d33f2ca83" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 36045 } } peers { permanent_uuid: "9ee0c011d7d94b7797586c886907a057" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 37805 } } }
I20260928 15:02:19.705572 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:19.707044 24568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:19.707760 24568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:19.711551 24568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34ec3c34b2e4b9592addbe3f10d6243: Generated new TSK 0
I20260928 15:02:19.711717 24568 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:02:19.735936 24379 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.734253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.1.252.254:38827)
0928 15:02:19.734300 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.734302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.735235 (+ 933us) server_negotiation.cc:407] Connection header received
0928 15:02:19.735343 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.735344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.735356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.735386 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.735837 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47856: BlockingRecv error: recv got EOF from 127.0.0.1:47856 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 15:02:19.737145 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.734980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.1.252.253:36045)
0928 15:02:19.735018 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.735020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.736462 (+ 1442us) server_negotiation.cc:407] Connection header received
0928 15:02:19.736620 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.736622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.736635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.736666 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.737081 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52870: BlockingRecv error: recv got EOF from 127.0.0.1:52870 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:02:19.737753 24532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.734625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.1.252.252:37805)
0928 15:02:19.734665 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.734667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.736330 (+ 1663us) server_negotiation.cc:407] Connection header received
0928 15:02:19.736732 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.736733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.736744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.736772 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.737695 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51246: BlockingRecv error: recv got EOF from 127.0.0.1:51246 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:02:19.740458 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.739567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.1.252.253:36045)
0928 15:02:19.739622 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.739624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.739847 (+ 223us) server_negotiation.cc:407] Connection header received
0928 15:02:19.739978 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.739979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.739999 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.740029 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.740410 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52876: BlockingRecv error: recv got EOF from 127.0.0.1:52876 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 15:02:19.743758 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.742960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.1.252.253:36045)
0928 15:02:19.742992 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.742994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.743216 (+ 222us) server_negotiation.cc:407] Connection header received
0928 15:02:19.743347 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.743348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.743357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.743387 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.743707 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52890: BlockingRecv error: recv got EOF from 127.0.0.1:52890 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 15:02:19.747802 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.747070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.1.252.253:36045)
0928 15:02:19.747103 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.747105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.747313 (+ 208us) server_negotiation.cc:407] Connection header received
0928 15:02:19.747404 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.747405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.747414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.747442 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.747756 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52894: BlockingRecv error: recv got EOF from 127.0.0.1:52894 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 15:02:19.753947 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.753092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.1.252.253:36045)
0928 15:02:19.753128 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.753130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.753351 (+ 221us) server_negotiation.cc:407] Connection header received
0928 15:02:19.753444 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.753445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.753455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.753485 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.753888 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52910: BlockingRecv error: recv got EOF from 127.0.0.1:52910 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 15:02:19.764263 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.763340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.1.252.253:36045)
0928 15:02:19.763381 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.763384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.763640 (+ 256us) server_negotiation.cc:407] Connection header received
0928 15:02:19.763744 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.763745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.763756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.763785 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.764194 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52918: BlockingRecv error: recv got EOF from 127.0.0.1:52918 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:02:19.783222 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.781877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.1.252.253:36045)
0928 15:02:19.782189 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.782191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.782490 (+ 299us) server_negotiation.cc:407] Connection header received
0928 15:02:19.782653 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.782655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.782668 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.782699 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.783162 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52922: BlockingRecv error: recv got EOF from 127.0.0.1:52922 (error 108)
Metrics: {"server-negotiator.queue_time_us":284}
W20260928 15:02:19.817883 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.816898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.1.252.253:36045)
0928 15:02:19.816946 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.816948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.817230 (+ 282us) server_negotiation.cc:407] Connection header received
0928 15:02:19.817338 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.817339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.817352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.817399 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.817815 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52938: BlockingRecv error: recv got EOF from 127.0.0.1:52938 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 15:02:19.884562 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:19.883661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.1.252.253:36045)
0928 15:02:19.883730 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:19.883732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:19.883810 (+ 78us) server_negotiation.cc:407] Connection header received
0928 15:02:19.883920 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:19.883922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:19.883934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:19.884038 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:19.884495 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52954: BlockingRecv error: recv got EOF from 127.0.0.1:52954 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 15:02:20.014603 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:20.013862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.1.252.253:36045)
0928 15:02:20.013937 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.013939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.013966 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:20.014089 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.014090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.014102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.014170 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:20.014522 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52956: BlockingRecv error: recv got EOF from 127.0.0.1:52956 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 15:02:20.272677 24455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:20.271944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.1.252.253:36045)
0928 15:02:20.272030 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.272032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.272077 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:02:20.272103 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.272104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.272121 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.272166 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:20.272586 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52970: BlockingRecv error: recv got EOF from 127.0.0.1:52970 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:02:20.642632 24546 catalog_manager.cc:1259] Loaded cluster ID: 82445cffbeb94f2d89be859d0eda3eb5
I20260928 15:02:20.642704 24546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: loading cluster ID for follower catalog manager: success
I20260928 15:02:20.643482 24546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: acquiring CA information for follower catalog manager: success
I20260928 15:02:20.643810 24546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ee0c011d7d94b7797586c886907a057: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:02:20.699434 24588 catalog_manager.cc:1259] Loaded cluster ID: 82445cffbeb94f2d89be859d0eda3eb5
I20260928 15:02:20.699522 24588 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: loading cluster ID for follower catalog manager: success
I20260928 15:02:20.700390 24588 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: acquiring CA information for follower catalog manager: success
I20260928 15:02:20.700717 24588 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5742a0d3900943468175847d33f2ca83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:02:20.788741 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:20.785964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.1.252.253:36045)
0928 15:02:20.786112 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.786114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.786124 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:20.786171 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.786172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.786223 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.786300 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:20.786527 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.787048 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.787936 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.788151 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.788207 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:20.788394 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:20.788508 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:20.788569 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:20.788606 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 15:02:20.788623 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260928 15:02:20.793370 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:20.790801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.1.252.252:37805)
0928 15:02:20.790938 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.790940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.790948 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:20.791040 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.791041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.791076 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.791148 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:20.791284 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.791800 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.792600 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.792797 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.792865 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:20.793015 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:20.793118 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:20.793185 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:20.793225 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 15:02:20.793244 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 15:02:20.798077 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:20.795382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.1.252.252:37805)
0928 15:02:20.795421 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.795422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.795570 (+ 148us) server_negotiation.cc:407] Connection header received
0928 15:02:20.795611 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.795612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.795623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.795669 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:20.795821 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.796545 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.797307 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.797599 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.797666 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:20.797816 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:20.797849 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:20.797896 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:20.797976 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:02:20.798004 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:02:20.798233 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:20.795101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.1.252.253:36045)
0928 15:02:20.795145 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.795147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.795323 (+ 176us) server_negotiation.cc:407] Connection header received
0928 15:02:20.795435 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.795437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.795450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.795480 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:20.795714 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.796289 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.797369 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.797660 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.797714 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:20.797918 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:20.797955 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:20.798017 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:20.798113 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 15:02:20.798140 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:02:20.798454 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:20.795094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.1.252.254:38827)
0928 15:02:20.795222 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:20.795224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:20.795251 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:02:20.795347 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:20.795348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:20.795397 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:20.795493 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:20.795650 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.796443 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.797435 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:20.797680 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:20.797737 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:20.798091 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:20.798199 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:20.798275 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:20.798333 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 15:02:20.798357 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 15:02:20.800125 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24307
I20260928 15:02:20.805747 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24383
I20260928 15:02:20.810798 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24458
2026-09-28T15:02:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2841 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:20 dist-test-slave-sf70 krb5kdc[24676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:20 dist-test-slave-sf70 krb5kdc[24676](info): set up 2 sockets
Sep 28 15:02:20 dist-test-slave-sf70 krb5kdc[24676](info): commencing operation
krb5kdc: starting...
W20260928 15:02:21.869849 2035 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/test-user.keytab.
Sep 28 15:02:21 dist-test-slave-sf70 krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.252.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
Entry for principal kudu/127.1.252.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab.
I20260928 15:02:21.992771 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35793
--webserver_interface=127.1.252.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:35793,127.1.252.253:37109,127.1.252.252:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:22.074378 24689 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:22.074635 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:22.074673 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:22.074733 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:22.075834 24689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:22.075901 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:22.075946 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:22.075974 24689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:22.075994 24689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:22.077451 24689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:35793,127.1.252.253:37109,127.1.252.252:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.254.keytab
--principal=kudu/127.1.252.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.254:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.252.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:22.077739 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:22.077963 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:22.082782 24697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:22.082813 24695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:22.082907 24689 server_base.cc:1034] running on GCE node
W20260928 15:02:22.082909 24694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:22.083336 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:22.083626 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:22.084802 24689 hybrid_clock.cc:648] HybridClock initialized: now 1790607742084788 us; error 44 us; skew 500 ppm
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:22.093376 24689 init.cc:377] Logged in from keytab as kudu/127.1.252.254@KRBTEST.COM (short username kudu)
I20260928 15:02:22.093775 24689 webserver.cc:460] Webserver started at http://127.1.252.254:37421/ using document root <none> and password file <none>
I20260928 15:02:22.094017 24689 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:22.094064 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:22.094225 24689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:22.095312 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data/instance:
uuid: "5c8f958c681841f5a7094d73c36fddd1"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "9310969c96d05023cc11e9a8708fa65b"
server_key_iv: "4555d5f7a67ae7215a148805d4c28935"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.095676 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal/instance:
uuid: "5c8f958c681841f5a7094d73c36fddd1"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "9310969c96d05023cc11e9a8708fa65b"
server_key_iv: "4555d5f7a67ae7215a148805d4c28935"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.097141 24689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:02:22.097889 24704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:22.098093 24689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:02:22.098186 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal
uuid: "5c8f958c681841f5a7094d73c36fddd1"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "9310969c96d05023cc11e9a8708fa65b"
server_key_iv: "4555d5f7a67ae7215a148805d4c28935"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.098258 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:22.111194 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:22.113646 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:22.113796 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:22.118198 24689 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.254:35793
I20260928 15:02:22.118234 24757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.254:35793 every 8 connection(s)
I20260928 15:02:22.118597 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:22.119635 24758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:22.121272 24758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:22.127658 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24689
I20260928 15:02:22.127800 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-0/wal/instance
I20260928 15:02:22.128060 2035 external_mini_cluster.cc:1428] Setting key b93abcb6bcfa7a09e63bc3825aa58c71
I20260928 15:02:22.131803 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.122133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.1.252.254:35793)
0928 15:02:22.122259 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.122264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.122297 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:02:22.122860 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.122862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.122883 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.122989 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.123540 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.124436 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.125297 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.125611 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.125725 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.127937 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.127955 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.127955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.127963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.129904 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.130418 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.130421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.130422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.130551 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.130835 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.130836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.130836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.131085 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.131155 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.131430 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.131607 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260928 15:02:22.131842 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.121942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35793 (local address 127.0.0.1:40290)
0928 15:02:22.122102 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.122160 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.122697 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.122995 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.122998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.123286 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.123430 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.123434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.124535 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.124536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.125199 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.125201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.125722 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.125723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.125788 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.126257 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.126264 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.127790 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.129990 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.129997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.130000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.130298 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.130638 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.130639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.130640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.130753 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.131162 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.131169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.131248 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.131351 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.131544 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260928 15:02:22.132700 24758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:22.133090 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.132945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35844)
0928 15:02:22.133001 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.133055 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:02:22.133250 24707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
I20260928 15:02:22.133709 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.133598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35856)
0928 15:02:22.133647 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.133679 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 15:02:22.133821 24758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:37109: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.252.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab.
Entry for principal kudu/127.1.252.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab.
I20260928 15:02:22.154739 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:37109
--webserver_interface=127.1.252.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab
--principal=kudu/127.1.252.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:35793,127.1.252.253:37109,127.1.252.252:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:22.183060 24758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } attempt: 1
I20260928 15:02:22.183552 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.183409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35864)
0928 15:02:22.183473 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.183523 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 15:02:22.183986 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.183823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35872)
0928 15:02:22.183910 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.183953 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:02:22.184134 24758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:37109: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
W20260928 15:02:22.236312 24765 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:22.236539 24765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:22.236557 24765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:22.236578 24765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:22.237723 24765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:22.237766 24765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:22.237780 24765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:22.237792 24765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:22.237805 24765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:22.239322 24765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:35793,127.1.252.253:37109,127.1.252.252:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.253.keytab
--principal=kudu/127.1.252.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.253:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.252.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:22.239593 24765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:22.239838 24765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:22.245026 24771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:22.245131 24770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:22.245080 24773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:22.245219 24765 server_base.cc:1034] running on GCE node
I20260928 15:02:22.245479 24765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:22.245779 24765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:22.246948 24765 hybrid_clock.cc:648] HybridClock initialized: now 1790607742246943 us; error 38 us; skew 500 ppm
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:22.249462 24765 init.cc:377] Logged in from keytab as kudu/127.1.252.253@KRBTEST.COM (short username kudu)
I20260928 15:02:22.249927 24765 webserver.cc:460] Webserver started at http://127.1.252.253:33347/ using document root <none> and password file <none>
I20260928 15:02:22.250159 24765 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:22.250228 24765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:22.250380 24765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:22.251418 24765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data/instance:
uuid: "8cbccb4af6404d3e903a8cc504a46ab5"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "5ff3aa51df2f86cbf7d28e7a37b4917c"
server_key_iv: "538a1a60a39b0f34eedfa9b4a8e4c6c5"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.251793 24765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal/instance:
uuid: "8cbccb4af6404d3e903a8cc504a46ab5"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "5ff3aa51df2f86cbf7d28e7a37b4917c"
server_key_iv: "538a1a60a39b0f34eedfa9b4a8e4c6c5"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.252297 24758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } attempt: 2
I20260928 15:02:22.252703 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.252549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35888)
0928 15:02:22.252617 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.252665 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 15:02:22.253144 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.253026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35896)
0928 15:02:22.253077 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.253108 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:02:22.253269 24758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.253:37109: Network error: Client connection negotiation failed: client connection to 127.1.252.253:37109: connect: Connection refused (error 111)
I20260928 15:02:22.253547 24765 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:02:22.254490 24780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:22.254726 24765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:22.254820 24765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal
uuid: "8cbccb4af6404d3e903a8cc504a46ab5"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "5ff3aa51df2f86cbf7d28e7a37b4917c"
server_key_iv: "538a1a60a39b0f34eedfa9b4a8e4c6c5"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.254892 24765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:22.268251 24765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:22.270190 24765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:22.270346 24765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:22.274484 24765 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.253:37109
I20260928 15:02:22.274521 24833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.253:37109 every 8 connection(s)
I20260928 15:02:22.274883 24765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/data/info.pb
I20260928 15:02:22.276049 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:22.277719 24834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:22.279567 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24765
I20260928 15:02:22.279697 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-1/wal/instance
I20260928 15:02:22.279985 2035 external_mini_cluster.cc:1428] Setting key 75d9807bf505ace1ddf8a4501d9ebb56
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:22.287952 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.278392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.1.252.254:35793)
0928 15:02:22.278472 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.278474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.278627 (+ 153us) server_negotiation.cc:407] Connection header received
0928 15:02:22.279125 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.279126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.279139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.279208 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.280074 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.280660 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.281744 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.282000 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.282115 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.284609 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.284611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.284611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.284620 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.286508 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.286906 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.286907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.286908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.286990 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.287231 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.287232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.287232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.287393 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.287475 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.287724 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.287820 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:02:22.288036 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.278330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35793 (local address 127.0.0.1:40306)
0928 15:02:22.278501 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.278540 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.279003 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.279283 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.279286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.279558 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.279974 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.279980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.280782 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.280783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.281645 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.281647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.282146 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.282148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.282303 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.282807 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.282818 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.284498 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.286604 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.286610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.286612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.286824 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.287063 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.287064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.287064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.287158 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.287469 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.287474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.287587 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.287669 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.287784 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260928 15:02:22.288853 24834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.252.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.252.252@KRBTEST.COM" created.
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:22.296819 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.289116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35906)
0928 15:02:22.289166 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.289176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.289211 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.289511 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.289512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.289627 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.289676 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.289678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.290630 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.290630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.291299 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.291302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.291781 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.291782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.291874 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.292295 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.292302 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.293552 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.295734 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.295735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.295735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.295956 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.296199 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.296199 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.296200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.296253 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.296604 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.296604 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.296641 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.296692 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.296759 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 15:02:22.297008 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.289198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.1.252.253:37109)
0928 15:02:22.289371 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.289374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.289389 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:22.289408 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.289410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.289423 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.289483 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.289749 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.290516 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.291383 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.291696 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.291755 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.293661 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.293668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.293669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.293675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.295644 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.296045 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.296049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.296049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.296120 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.296323 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.296324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.296324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.296529 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.296633 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.296732 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.296826 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260928 15:02:22.297678 24834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:22.298022 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.297905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42848)
0928 15:02:22.297955 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.298004 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:02:22.298187 24783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
I20260928 15:02:22.298699 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.298611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42856)
0928 15:02:22.298659 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.298683 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:02:22.298823 24834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:36521: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.252.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab.
Entry for principal kudu/127.1.252.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab.
I20260928 15:02:22.311589 2035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWmXXpN/build/release/bin/kudu
/tmp/dist-test-taskWmXXpN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:36521
--webserver_interface=127.1.252.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab
--principal=kudu/127.1.252.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.252.212:45841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.252.254:35793,127.1.252.253:37109,127.1.252.252:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:02:22.344425 24758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } attempt: 3
I20260928 15:02:22.348070 24834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } attempt: 1
I20260928 15:02:22.348495 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.348347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42858)
0928 15:02:22.348424 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.348465 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:02:22.348901 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.348792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42860)
0928 15:02:22.348841 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.348869 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:02:22.349025 24834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:36521: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:22.352720 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.344919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.1.252.253:37109)
0928 15:02:22.344980 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.344983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.345032 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:02:22.345062 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.345063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.345074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.345127 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.345446 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.346227 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.347034 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.347420 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.347506 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.349920 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.349921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.349922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.349928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.351550 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.351977 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.351981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.351981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.352043 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.352249 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.352249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.352249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.352382 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.352440 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.352597 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.352661 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:02:22.352767 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.344814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35912)
0928 15:02:22.344886 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.344899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.344976 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.345142 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.345143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.345287 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.345364 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.345366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.346374 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.346375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.346943 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.346945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.347524 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.347526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.347601 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.348341 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.348348 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.349831 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.351635 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.351636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.351637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.351862 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.352116 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.352117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.352117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.352183 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.352447 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.352448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.352500 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.352579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.352673 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 15:02:22.353293 24758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:02:22.353624 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.353541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42870)
0928 15:02:22.353574 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.353603 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 15:02:22.353901 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.353845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42874)
0928 15:02:22.353868 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.353888 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 15:02:22.353986 24758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:36521: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
I20260928 15:02:22.385141 24758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } attempt: 1
I20260928 15:02:22.385643 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.385472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42888)
0928 15:02:22.385575 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.385613 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:02:22.385995 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.385900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42898)
0928 15:02:22.385946 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.385973 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 15:02:22.386083 24758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:36521: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
W20260928 15:02:22.395171 24841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:02:22.395370 24841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:22.395387 24841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:22.395407 24841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:22.396559 24841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:22.396596 24841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:22.396611 24841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:22.396623 24841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:22.396659 24841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:22.398115 24841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.252.212:45841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.252.254:35793,127.1.252.253:37109,127.1.252.252:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/krb5kdc/kudu_127.1.252.252.keytab
--principal=kudu/127.1.252.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.252.252:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.252.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:22.398376 24841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:22.398608 24841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:22.403498 24846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:22.403496 24849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:22.403527 24841 server_base.cc:1034] running on GCE node
W20260928 15:02:22.403529 24847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:22.404031 24841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:22.404345 24841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:22.405524 24841 hybrid_clock.cc:648] HybridClock initialized: now 1790607742405502 us; error 32 us; skew 500 ppm
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:22.408128 24841 init.cc:377] Logged in from keytab as kudu/127.1.252.252@KRBTEST.COM (short username kudu)
I20260928 15:02:22.408504 24841 webserver.cc:460] Webserver started at http://127.1.252.252:42999/ using document root <none> and password file <none>
I20260928 15:02:22.408743 24841 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:22.408788 24841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:22.408944 24841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:22.409998 24841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data/instance:
uuid: "d8a35e01eed1436391b0fa248b239335"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "a7c08ce7355a2c1df4c94f9ce0570ef2"
server_key_iv: "ec42212e7c0ad2530c200eb3a9332c07"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.410382 24841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal/instance:
uuid: "d8a35e01eed1436391b0fa248b239335"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "a7c08ce7355a2c1df4c94f9ce0570ef2"
server_key_iv: "ec42212e7c0ad2530c200eb3a9332c07"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.411698 24841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:02:22.412500 24856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:22.412686 24841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:22.412786 24841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data,/tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal
uuid: "d8a35e01eed1436391b0fa248b239335"
format_stamp: "Formatted at 2026-09-28 15:02:22 on dist-test-slave-sf70"
server_key: "a7c08ce7355a2c1df4c94f9ce0570ef2"
server_key_iv: "ec42212e7c0ad2530c200eb3a9332c07"
server_key_version: "encryptionkey@0"
I20260928 15:02:22.412875 24841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:22.417176 24834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } attempt: 2
I20260928 15:02:22.417589 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.417465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42914)
0928 15:02:22.417525 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.417563 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:02:22.417932 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.417834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42918)
0928 15:02:22.417877 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.417915 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 15:02:22.418011 24834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.252.252:36521: Network error: Client connection negotiation failed: client connection to 127.1.252.252:36521: connect: Connection refused (error 111)
I20260928 15:02:22.421172 24841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:22.423149 24841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:22.423307 24841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:22.427326 24841 rpc_server.cc:307] RPC server started. Bound to: 127.1.252.252:36521
I20260928 15:02:22.427356 24908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.252.252:36521 every 8 connection(s)
I20260928 15:02:22.427693 24841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/data/info.pb
I20260928 15:02:22.428745 24909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:22.430275 24909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:22.436342 2035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWmXXpN/build/release/bin/kudu as pid 24841
I20260928 15:02:22.436460 2035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWmXXpN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607474232031-2035-0/minicluster-data/master-2/wal/instance
I20260928 15:02:22.436674 2035 external_mini_cluster.cc:1428] Setting key 8deaa6cd1f700637dee365b6ca7d24d8
I20260928 15:02:22.439587 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.430972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.1.252.254:35793)
0928 15:02:22.431033 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.431035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.431220 (+ 185us) server_negotiation.cc:407] Connection header received
0928 15:02:22.431735 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.431735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.431745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.431804 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.432434 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.432908 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.433844 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.434034 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.434090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.436221 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.436222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.436222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.436230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.438201 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.438612 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.438613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.438614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.438677 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.438912 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.438913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.438913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.439066 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.439136 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.439458 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.439524 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:02:22.439823 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.430926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.254:35793 (local address 127.0.0.1:40314)
0928 15:02:22.431082 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.431121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.431642 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.431862 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.431865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.432157 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.432346 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.432350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.432990 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.432991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.433764 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.433767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.434125 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.434126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.434226 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.434735 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.434744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.436114 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.438294 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.438301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.438303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.438533 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.438744 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.438745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.438745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.438848 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.439139 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.439146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.439314 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.439429 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.439605 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260928 15:02:22.440412 24909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:22.446008 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.437065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.1.252.252:36521)
0928 15:02:22.437197 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.437202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.437296 (+ 94us) server_negotiation.cc:407] Connection header received
0928 15:02:22.437351 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.437354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.437368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.437462 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.438013 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.439001 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.440114 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.440558 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.440635 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.442907 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.442915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.442916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.442922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.444555 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.445174 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.445177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.445177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.445240 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.445429 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.445430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.445430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.445628 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.445773 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.445827 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.445903 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260928 15:02:22.447355 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.440652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.253:37109 (local address 127.0.0.1:35920)
0928 15:02:22.440701 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.440711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.440742 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.440919 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.440920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.441082 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.441135 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.441138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.441824 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.441825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.442337 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.442339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.442824 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.442825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.442897 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.443219 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.443223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.444580 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.446356 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.446357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.446357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.446526 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.446749 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.446749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.446750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.446817 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.447113 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.447113 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.447156 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.447231 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.447296 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 15:02:22.447489 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.440723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.1.252.253:37109)
0928 15:02:22.440777 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.440779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.440787 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:22.440813 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.440814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.440825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.440914 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.441264 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.441729 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.442447 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.442737 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.442830 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.444680 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.444681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.444681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.444687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.446273 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.446604 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.446606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.446606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.446677 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.446892 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.446893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.446893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.447042 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.447113 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.447281 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.447364 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:02:22.447821 24909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.252@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.254@KRBTEST.COM
I20260928 15:02:22.454898 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.446843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.1.252.254:35793)
0928 15:02:22.446886 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.446888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.446964 (+ 76us) server_negotiation.cc:407] Connection header received
0928 15:02:22.447044 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.447044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.447053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.447123 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.447721 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.448455 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.449113 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.449319 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.449402 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.452151 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.452152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.452152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.452159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.453676 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.454081 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.454082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.454082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.454157 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.454357 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.454357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.454358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.454497 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.454663 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.454725 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.454802 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:02:22.454927 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.448083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42946)
0928 15:02:22.448137 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.448148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.448187 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.448314 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.448314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.448420 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.448447 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.448448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.449053 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.449054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.449515 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.449517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.449913 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.449914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.449967 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.450340 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.450344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.451398 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.453791 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.453792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.453792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.453992 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.454224 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.454225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.454225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.454288 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.454578 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.454579 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.454623 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.454699 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.454794 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 15:02:22.454979 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.448153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.1.252.252:36521)
0928 15:02:22.448187 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.448189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.448196 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:02:22.448235 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.448235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.448244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.448311 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.448514 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.448972 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.449618 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.449830 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.449912 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.451496 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.451497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.451497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.451503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.453145 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.454081 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.454082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.454082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.454156 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.454357 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.454357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.454358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.454505 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.454567 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.454736 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.454845 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":17}
I20260928 15:02:22.456404 24909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: Bootstrap starting.
I20260928 15:02:22.457263 24909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:22.457609 24909 log.cc:824] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:22.458477 24909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: No bootstrap required, opened a new log
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.252.253@KRBTEST.COM
I20260928 15:02:22.460063 24909 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.460245 24909 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a35e01eed1436391b0fa248b239335, State: Initialized, Role: FOLLOWER
I20260928 15:02:22.460331 24909 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.460677 24909 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:22.460650 24915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.460759 24915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:22.461220 24758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } attempt: 2
I20260928 15:02:22.463272 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.455522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.1.252.253:37109)
0928 15:02:22.455565 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.455567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.455582 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:22.455612 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.455613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.455622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.455666 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.456211 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.456731 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.457434 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.457780 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.457851 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.460121 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.460123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.460123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.460130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.461975 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.462349 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.462350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.462350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.462399 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.462615 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.462616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.462616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.462746 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.462896 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.463103 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.463181 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:02:22.465003 2035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 15:02:22.464489 24926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:22.465111 24926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.254@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:02:22.464249 24841 master_runner.cc:186] Error getting master registration for 127.1.252.254:35793: OK, instance_id { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" instance_seqno: 1790607742116924 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:22.467141 24841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:22.466475 24731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c8f958c681841f5a7094d73c36fddd1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322
I20260928 15:02:22.469554 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.461525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.1.252.252:36521)
0928 15:02:22.461565 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.461567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.461577 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:22.461599 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.461600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.461609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.461686 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.461972 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.462669 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.463693 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.463984 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.464056 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.466562 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.466563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.466564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.466571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.468290 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.468711 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.468712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.468713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.468775 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.468984 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.468985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.468985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.469151 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.469216 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.469384 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.469460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:02:22.469559 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.461465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42948)
0928 15:02:22.461509 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.461519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.461556 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.461695 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.461695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.461859 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.461901 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.461904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.462743 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.462744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.463508 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.463511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.464109 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.464110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.464165 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.464533 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.464539 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.466464 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.468379 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.468380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.468380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.468618 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.468845 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.468846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.468847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.468918 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.469223 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.469224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.469272 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.469351 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.469451 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 15:02:22.471232 24758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1: Bootstrap starting.
I20260928 15:02:22.472013 24758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:22.472329 24758 log.cc:824] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:22.473187 24758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1: No bootstrap required, opened a new log
I20260928 15:02:22.474630 24758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.474769 24758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8f958c681841f5a7094d73c36fddd1, State: Initialized, Role: FOLLOWER
I20260928 15:02:22.474851 24758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.475154 24758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:22.475116 24929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.475205 24731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:02:22.475204 24929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:22.475282 24731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:22.476068 24731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.476300 24731 leader_election.cc:288] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [CANDIDATE]: Term 1 election: Requested vote from peers 8cbccb4af6404d3e903a8cc504a46ab5 (127.1.252.253:37109), d8a35e01eed1436391b0fa248b239335 (127.1.252.252:36521)
I20260928 15:02:22.476591 24883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c8f958c681841f5a7094d73c36fddd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d8a35e01eed1436391b0fa248b239335"
I20260928 15:02:22.476742 24883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:22.477701 24883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8f958c681841f5a7094d73c36fddd1 in term 1.
I20260928 15:02:22.477916 24707 leader_election.cc:302] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8f958c681841f5a7094d73c36fddd1, d8a35e01eed1436391b0fa248b239335; no voters:
I20260928 15:02:22.478052 24929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:22.478217 24929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [term 1 LEADER]: Becoming Leader. State: Replica: 5c8f958c681841f5a7094d73c36fddd1, State: Running, Role: LEADER
I20260928 15:02:22.478324 24929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.478930 24941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c8f958c681841f5a7094d73c36fddd1. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.479043 24941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:22.480507 24940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:22.480592 24940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:22.477833 24689 master_runner.cc:186] Error getting master registration for 127.1.252.253:37109: OK, instance_id { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" instance_seqno: 1790607742273364 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:02:22.480682 24943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:22.480712 24689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:02:22.480852 24943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:22.482798 24883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [term 1 FOLLOWER]: Refusing update from remote peer 5c8f958c681841f5a7094d73c36fddd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:22.483072 24929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:22.484303 24915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c8f958c681841f5a7094d73c36fddd1. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.484396 24915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:22.493052 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.481876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.1.252.254:35793)
0928 15:02:22.481914 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.481916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.482042 (+ 126us) server_negotiation.cc:407] Connection header received
0928 15:02:22.482143 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.482144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.482156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.482185 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.485074 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.485817 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.486866 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.487143 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.487186 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.489522 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.489524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.489524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.489531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.491211 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.492166 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.492167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.492168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.492230 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.492465 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.492466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.492467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.492605 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.492689 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.492884 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.492964 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:02:22.494429 24949 mvcc.cc:205] Tried to move back new op lower bound from 7334329313206059008 to 7334329313191940096. Current Snapshot: MvccSnapshot[applied={T|T < 7334329313206059008}]
I20260928 15:02:22.494951 24915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.495042 24915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:22.495193 24929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.495298 24929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:22.495376 24951 mvcc.cc:205] Tried to move back new op lower bound from 7334329313206059008 to 7334329313191940096. Current Snapshot: MvccSnapshot[applied={T|T < 7334329313206059008}]
I20260928 15:02:22.496496 24943 catalog_manager.cc:1347] Generated new cluster ID: c2c2bec52a7a4818950ca6db50e09a1b
I20260928 15:02:22.496562 24943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:22.500095 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.489026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.1.252.253:37109)
0928 15:02:22.489083 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.489085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.489372 (+ 287us) server_negotiation.cc:407] Connection header received
0928 15:02:22.489541 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.489543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.489554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.489585 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.490655 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.491367 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.493089 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.493405 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.493445 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.496744 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.496745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.496746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.496764 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.498351 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.498999 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.499000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.499000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.499061 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.499366 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.499367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.499367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.499515 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.499585 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.499923 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.500001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:02:22.509123 24834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } attempt: 3
I20260928 15:02:22.512041 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.489378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.1.252.252:36521)
0928 15:02:22.489420 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.489422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.495602 (+ 6180us) server_negotiation.cc:407] Connection header received
0928 15:02:22.495662 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.495663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.495675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.495740 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.496336 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.497033 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.498051 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.498362 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.498414 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.508844 (+ 10430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.508845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.508845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.508853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.510563 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.510955 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.510955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.510956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.511025 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.511231 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.511232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.511232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.511378 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.511489 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.511850 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.511938 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:02:22.512424 24943 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:22.513149 24943 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 15:02:22 dist-test-slave-sf70 krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.252.253@KRBTEST.COM for kudu/127.1.252.252@KRBTEST.COM
I20260928 15:02:22.521471 24943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1: Generated new TSK 0
I20260928 15:02:22.521660 24943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:22.521719 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.509357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.252.252:36521 (local address 127.0.0.1:42968)
0928 15:02:22.509489 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:22.509500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:22.509540 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:22.510859 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:22.510860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:22.511027 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:22.511093 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.511095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.513122 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.513123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.513932 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:22.513934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.514838 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.514839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:22.514919 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.515373 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:22.515378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:22.517015 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:22.520289 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.520290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.520291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.520534 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.520903 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:22.520904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:22.520905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:22.520977 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:22.521412 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:22.521412 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:22.521469 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:02:22.521524 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 15:02:22.521619 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 15:02:22.522363 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:22.509623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.1.252.252:36521)
0928 15:02:22.509857 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.509858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.509866 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:22.510687 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.510688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.510700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.511183 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.511254 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.513007 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.514150 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:22.514695 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:22.515076 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:22.517124 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:22.517126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:22.517126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:22.517132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:22.520146 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.520682 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.520683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.520683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.520775 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:22.521061 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:22.521062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:22.521063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:22.521310 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:22.522090 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:22.522160 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 15:02:22.522239 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":109,"threads_started":1}
I20260928 15:02:22.523792 24834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: Bootstrap starting.
I20260928 15:02:22.524551 24834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:22.524859 24834 log.cc:824] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:22.526580 24834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: No bootstrap required, opened a new log
I20260928 15:02:22.529238 24834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.529547 24834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbccb4af6404d3e903a8cc504a46ab5, State: Initialized, Role: FOLLOWER
I20260928 15:02:22.529747 24834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } }
I20260928 15:02:22.530496 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c8f958c681841f5a7094d73c36fddd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8cbccb4af6404d3e903a8cc504a46ab5"
I20260928 15:02:22.530557 24806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:22.530434 24834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:22.531399 24806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [term 1 FOLLOWER]: Refusing update from remote peer 5c8f958c681841f5a7094d73c36fddd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:22.531729 24929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:22.532752 24807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8f958c681841f5a7094d73c36fddd1 in term 1.
I20260928 15:02:22.533337 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.533910 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:22.534955 24974 mvcc.cc:205] Tried to move back new op lower bound from 7334329313361526784 to 7334329313191940096. Current Snapshot: MvccSnapshot[applied={T|T < 7334329313206059008}]
I20260928 15:02:22.535728 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c8f958c681841f5a7094d73c36fddd1. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.536053 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:02:22.536146 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.536252 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:02:22.536401 24972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:22.536463 24972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 15:02:22.568620 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.564422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.1.252.253:37109)
0928 15:02:22.564484 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.564486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.567826 (+ 3340us) server_negotiation.cc:407] Connection header received
0928 15:02:22.567975 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.567976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.567989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.568023 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.568544 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35952: BlockingRecv error: recv got EOF from 127.0.0.1:35952 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 15:02:22.569442 24761 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.564233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.1.252.254:35793)
0928 15:02:22.564288 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.564291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.568832 (+ 4541us) server_negotiation.cc:407] Connection header received
0928 15:02:22.568936 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.568937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.568950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.568978 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.569350 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40334: BlockingRecv error: recv got EOF from 127.0.0.1:40334 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 15:02:22.570173 24957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.564885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.1.252.252:36521)
0928 15:02:22.564926 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.564928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.569678 (+ 4750us) server_negotiation.cc:407] Connection header received
0928 15:02:22.569770 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.569771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.569782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.569810 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.570116 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42976: BlockingRecv error: recv got EOF from 127.0.0.1:42976 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 15:02:22.572798 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.572009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.1.252.253:37109)
0928 15:02:22.572058 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.572059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.572280 (+ 221us) server_negotiation.cc:407] Connection header received
0928 15:02:22.572385 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.572386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.572404 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.572444 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.572750 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35964: BlockingRecv error: recv got EOF from 127.0.0.1:35964 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 15:02:22.575975 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.575111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.1.252.253:37109)
0928 15:02:22.575147 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.575149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.575406 (+ 257us) server_negotiation.cc:407] Connection header received
0928 15:02:22.575531 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.575532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.575543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.575574 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.575932 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35968: BlockingRecv error: recv got EOF from 127.0.0.1:35968 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 15:02:22.579879 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.579121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.1.252.253:37109)
0928 15:02:22.579155 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.579157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.579370 (+ 213us) server_negotiation.cc:407] Connection header received
0928 15:02:22.579484 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.579485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.579496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.579525 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.579836 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35976: BlockingRecv error: recv got EOF from 127.0.0.1:35976 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 15:02:22.585858 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.585055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.1.252.253:37109)
0928 15:02:22.585089 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.585091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.585336 (+ 245us) server_negotiation.cc:407] Connection header received
0928 15:02:22.585447 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.585448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.585459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.585489 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.585813 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35978: BlockingRecv error: recv got EOF from 127.0.0.1:35978 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 15:02:22.596096 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.595109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.1.252.253:37109)
0928 15:02:22.595149 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.595151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.595412 (+ 261us) server_negotiation.cc:407] Connection header received
0928 15:02:22.595533 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.595534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.595545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.595575 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.596032 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35988: BlockingRecv error: recv got EOF from 127.0.0.1:35988 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:02:22.614578 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.613587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.1.252.253:37109)
0928 15:02:22.613632 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.613635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.613928 (+ 293us) server_negotiation.cc:407] Connection header received
0928 15:02:22.614041 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.614042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.614055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.614085 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.614516 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35994: BlockingRecv error: recv got EOF from 127.0.0.1:35994 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:02:22.648756 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.647865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.1.252.253:37109)
0928 15:02:22.647933 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.647936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.648096 (+ 160us) server_negotiation.cc:407] Connection header received
0928 15:02:22.648177 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.648178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.648191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.648302 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.648674 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35998: BlockingRecv error: recv got EOF from 127.0.0.1:35998 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 15:02:22.714715 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.713979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.1.252.253:37109)
0928 15:02:22.714031 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.714033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.714128 (+ 95us) server_negotiation.cc:407] Connection header received
0928 15:02:22.714197 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.714198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.714210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.714266 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.714655 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36006: BlockingRecv error: recv got EOF from 127.0.0.1:36006 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 15:02:22.844725 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:22.843868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.1.252.253:37109)
0928 15:02:22.843958 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:22.843961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:22.844081 (+ 120us) server_negotiation.cc:407] Connection header received
0928 15:02:22.844192 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:22.844193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:22.844205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:22.844286 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:22.844657 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36022: BlockingRecv error: recv got EOF from 127.0.0.1:36022 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:02:22.988556 24929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c8f958c681841f5a7094d73c36fddd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8f958c681841f5a7094d73c36fddd1" member_type: VOTER last_known_addr { host: "127.1.252.254" port: 35793 } } peers { permanent_uuid: "8cbccb4af6404d3e903a8cc504a46ab5" member_type: VOTER last_known_addr { host: "127.1.252.253" port: 37109 } } peers { permanent_uuid: "d8a35e01eed1436391b0fa248b239335" member_type: VOTER last_known_addr { host: "127.1.252.252" port: 36521 } } }
I20260928 15:02:22.988724 24929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8f958c681841f5a7094d73c36fddd1 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:23.102720 24838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:23.102076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.1.252.253:37109)
0928 15:02:23.102163 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.102165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.102174 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:23.102209 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.102209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.102219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.102280 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:23.102672 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36032: BlockingRecv error: recv got EOF from 127.0.0.1:36032 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:02:23.465561 24926 catalog_manager.cc:1259] Loaded cluster ID: c2c2bec52a7a4818950ca6db50e09a1b
I20260928 15:02:23.465677 24926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: loading cluster ID for follower catalog manager: success
I20260928 15:02:23.466511 24926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: acquiring CA information for follower catalog manager: success
I20260928 15:02:23.466846 24926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8a35e01eed1436391b0fa248b239335: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:02:23.536902 24972 catalog_manager.cc:1259] Loaded cluster ID: c2c2bec52a7a4818950ca6db50e09a1b
I20260928 15:02:23.536979 24972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: loading cluster ID for follower catalog manager: success
I20260928 15:02:23.537853 24972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: acquiring CA information for follower catalog manager: success
I20260928 15:02:23.538128 24972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8cbccb4af6404d3e903a8cc504a46ab5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:02:23.618729 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:23.616078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.1.252.253:37109)
0928 15:02:23.616212 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.616214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.616226 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:23.616267 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.616267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.616324 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.616454 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:23.616630 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.617158 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.617952 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.618155 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.618220 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:23.618395 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:23.618519 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:23.618595 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:23.618638 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 15:02:23.618659 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 15:02:23.623473 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:23.620996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.1.252.252:36521)
0928 15:02:23.621110 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.621112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.621127 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:23.621165 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.621166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.621204 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.621274 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:23.621382 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.621866 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.622578 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.622764 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.622813 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:23.623011 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:23.623112 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:23.623185 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:23.623267 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 15:02:23.623286 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 15:02:23.628360 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:23.625192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.1.252.253:37109)
0928 15:02:23.625235 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.625237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.625401 (+ 164us) server_negotiation.cc:407] Connection header received
0928 15:02:23.625499 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.625501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.625512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.625553 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:23.625695 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.626426 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.627542 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.627763 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.627812 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:23.628078 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:23.628108 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:23.628158 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:23.628255 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 15:02:23.628280 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:02:23.628676 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:23.625174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.1.252.254:35793)
0928 15:02:23.625270 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.625272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.625463 (+ 191us) server_negotiation.cc:407] Connection header received
0928 15:02:23.625517 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.625518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.625567 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.625643 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:23.625772 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.626517 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.627246 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.627464 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.627546 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:23.628054 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:23.628157 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:23.628216 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:23.628572 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 15:02:23.628597 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":52,"threads_started":1}
I20260928 15:02:23.628895 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:23.625519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.1.252.252:36521)
0928 15:02:23.625555 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:23.625556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:23.625817 (+ 261us) server_negotiation.cc:407] Connection header received
0928 15:02:23.625849 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:23.625850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:23.625860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:23.625923 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:23.626056 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.626549 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.627969 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:23.628289 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:23.628364 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:23.628541 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:23.628571 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:23.628614 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:23.628693 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 15:02:23.628848 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 15:02:23.630285 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24689
I20260928 15:02:23.636091 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24765
I20260928 15:02:23.641084 2035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWmXXpN/build/release/bin/kudu with pid 24841
2026-09-28T15:02:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2830 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5672 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269418 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST